play_ Posted January 11, 2007 Share Posted January 11, 2007 I'm trying to build my own syntax hiliter, and someone suggested i use tokenizers instead of regexps to hilite code.Can anyone point me to any direction on this?Totally lost here. Not even 100% sure what tokenizers are.I have here a list of php tokens [url=http://dk2.php.net/manual/en/tokens.php]http://dk2.php.net/manual/en/tokens.php[/url].. but what would i do with it? Link to comment https://forums.phpfreaks.com/topic/33724-tokens/ Share on other sites More sharing options...
HuggieBear Posted January 11, 2007 Share Posted January 11, 2007 Also look here for a reference...http://dk2.php.net/manual/en/ref.tokenizer.phpRegardsHuggie Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-158122 Share on other sites More sharing options...
play_ Posted January 11, 2007 Author Share Posted January 11, 2007 How can i use that to hilight other language's tokens? Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-158162 Share on other sites More sharing options...
play_ Posted January 11, 2007 Author Share Posted January 11, 2007 I think i somewhat understand....reading the example from here: [url=http://dk2.php.net/manual/en/function.token-get-all.php]http://dk2.php.net/manual/en/function.token-get-all.php[/url]But how can i highlite other language's tokens? Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-158163 Share on other sites More sharing options...
HuggieBear Posted January 11, 2007 Share Posted January 11, 2007 I think this is maybe a job for Google...RegardsHuggie Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-158166 Share on other sites More sharing options...
play_ Posted January 12, 2007 Author Share Posted January 12, 2007 I've been searching. There isn't much on the topic Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-158998 Share on other sites More sharing options...
redbullmarky Posted January 12, 2007 Share Posted January 12, 2007 would the [url=http://uk2.php.net/manual/en/function.highlight-string.php]highlight_string[/url] function help for what you're trying to do? Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159036 Share on other sites More sharing options...
play_ Posted January 12, 2007 Author Share Posted January 12, 2007 [quote author=redbullmarky link=topic=121916.msg503047#msg503047 date=1168606551]would the [url=http://uk2.php.net/manual/en/function.highlight-string.php]highlight_string[/url] function help for what you're trying to do?[/quote]Nope.It would if i was only hilighting php code.However, since it's a pastebin, users will be submitting different language's codes.Therefore i have to create my own method for hiliting.I was going to go with regexps, but someone suggested using a tokenizer instead, but i can't find much info on it Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159042 Share on other sites More sharing options...
redbullmarky Posted January 12, 2007 Share Posted January 12, 2007 to be honest, this probably IS a job for regular expressions, not the tokenizer - especially as you state that it'll not just be PHP code that's highlighted. the tokenizer functions mentioned previously here are relevent to PHP tokens only and not, for example, Perl, C, ASP, etc, etc. Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159043 Share on other sites More sharing options...
play_ Posted January 12, 2007 Author Share Posted January 12, 2007 Yes. But i was thinking there is a way i could define my own tokensfor example, T_OPEN_TAG is <?Perhaps i could change the value of T_OPEN_TAG to better suit each languageAnd do the same with strings, comments, etc Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159069 Share on other sites More sharing options...
redbullmarky Posted January 12, 2007 Share Posted January 12, 2007 a google search for '[url=http://www.google.co.uk/search?client=firefox-a&rls=org.mozilla%3Aen-GB%3Aofficial&channel=s&hl=en&q=syntax+highlight+php&meta=&btnG=Google+Search]syntax highlight php[/url]' came up with:http://qbnz.com/highlighter/you might want to look at some of the other results there, too... Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159087 Share on other sites More sharing options...
play_ Posted January 12, 2007 Author Share Posted January 12, 2007 Yes.that is possibly the most popular syntax hiliter out there, GeSHI!a bit too complex for me though =/ Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159097 Share on other sites More sharing options...
play_ Posted January 12, 2007 Author Share Posted January 12, 2007 bumping it. Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159511 Share on other sites More sharing options...
ShogunWarrior Posted January 12, 2007 Share Posted January 12, 2007 I'm not sure what you mean that GeSHI is too complicated for you.I use GeSHI because it's such an easy snap-in solution for highlighting and does a great job.I'd steer around writing a language parser altogether! Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159518 Share on other sites More sharing options...
play_ Posted January 12, 2007 Author Share Posted January 12, 2007 I didn't mean GeSHI is hard to use, i meant it's code is a little advanced for me.I could use it on my site, but i don't want to. I want to create my own :)Although not as complex as GeSHI, just want to hilight keywords, comments and strings. (and perhaps ints/floats) Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159520 Share on other sites More sharing options...
redbullmarky Posted January 12, 2007 Share Posted January 12, 2007 ok, so eliminating GeSHI for being too complex, and php's tokenizer for being PHP-only, it's regex all the way... Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159522 Share on other sites More sharing options...
play_ Posted January 12, 2007 Author Share Posted January 12, 2007 Alright then.Can't swerve from regexps any longer :( Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159529 Share on other sites More sharing options...
ShogunWarrior Posted January 12, 2007 Share Posted January 12, 2007 Ah, don't frett. Regexps are great when you get to know them. You can write a little regex and it does all the work for you.Go download the Regex Coach: http://weitz.de/regex-coach/#install and you can test your regexps live and adjust them as you go. Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159530 Share on other sites More sharing options...
play_ Posted January 12, 2007 Author Share Posted January 12, 2007 Thanks Shogun. :) Link to comment https://forums.phpfreaks.com/topic/33724-tokens/#findComment-159536 Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.