Over on Reddit somebody asked: "Why is command prompt still such a big part of Linux?"

And I answered it with:

There is a major difference between the command prompt as found on Windows and the one generally found on Linux and other Unix systems.

The one found on Windows is generally severely under-featured with fixed sized windows and poor selection of command-line tools to make it appealing to use it. Even other tools that extend what is often available on Windows generally do not try they hardest to match what is available elsewhere, and there can be a mismatch between what such tools provide on Windows and what Windows in general would expect, so tools like Cygwin can only do so much even with terminals such as Poderosa.

Windows by making it un-worthwhile to pursue command-line tools, creates a heavy demand for GUI tools which then end up becoming too many and hard to use together. Also Windows does not ship with common tools, so users are left to hunt for their own tools and it introduces them to potential security nightmares of downloading executables from untrustworthy parties and once they become tired of such things they end up trying to pick one of the standard tools which can lose in features to some other offerings found in the market. The command-line tends to keep things more straightforward with tools that ship in a standardized way. For example, "tar" and its compacting companions "z", "j"...

Also, one of such tools that come in handy quite often is SSH for either terminal or for sending and receiving files in a secure way.

Besides, instead of creating a full-blown GUI application to automate a simple administrative task, creating a command-line tool is often quite straightforward in comparison.

And on Unix systems and on Linux in special, when folks want a graphical UI, they have tended to create a Web application. Such administrative applications often came in Perl and PHP flavors but who knows what tools folks have been using.

And having this focus on the command-line means that the server does not need to have a full-blown graphical user interface so it becomes easier to update its packages with upgrades and less demanding to keep things up-to-date with security and so on. (Fewer files.)

In between that, the truth lies.

Over on Reddit somebody asked: "Is it hopeless to program without a CS degree? "

And I answered it with:

Creating with programming is hard. Even though I do not have a degree, every now and again I come across some article or idea of a professional programmer who works at places like Microsoft which by sharing some of my own pain reminds me of how we are all humans after all, in that when creating something from scratch, however trivial it may seem on the surface of it, it can always be a real pain, shared by both real professionals and wannabes.

For example, I have been fighting over the past few days with creating undo/redo for my Text Editor, and one Microsoft guy once talked about how error-prone even creating a good undo/redo can be.

I also created a blog of my own (and lots of other things of my own, actually), and know how tough it can be to maintain it.

It is always easier to create something without actually having to use it then it is to create something and have to use it while you try to evolve it. Good data can become obsolete, you may have to go back and change some of it once you get tired of how something turned out to be and you update it.

Perhaps that can give you an idea of why in software development folks say that there is not a "Silver Bullet", only hard work, and years can go by faster than you can dream of it.

Somebody over on Reddit asked: "Really learning to program "

And I answered it with:

Programming has its formal side and its informal complement. The formal parts often can be taught in books and in college, but it can be quite detached from the needs of an average programmer for his day to day work in general.

I think that often programmers leverage the network effect of having better programmers than themselves to look after them. So by going to the right college and getting in touch with the right people can get one into the right workplace later on to continue his learning. Learning what exactly? A bit of everything and a much of a few things to develop one's specialty.

Say PHP and some PHP applications, a little of MySQL, browsers, Javascript, HTML, CSS, a little of Flash and a little more of image editing (Photoshop) make most of your specialty. You cannot quite change that any more. It's like one of the grandpas of Computer Science said in some kind of analogy, that the programmers who started their lives by first learning BASIC were damaged for life because BASIC (trying to explain it) was like almost nothing of what some folks deemed appreciable in a formal education and folks by specializing in BASIC and its needs were on the wrong track already...

So, what does that get us to? I for one. I started programming in my early teens in one of those old little computers which understood BASIC or some variation of it. The programs came in books and magazines and I had to type them in to then be able to run them, and the bigger ones almost never worked due to some kind of typo either in the writing or introduced by me or my dictator companion. ;-)

After my early teens I got through a period of no-programming but the seed had been planted in me very early on. In my late teens my brother got into the crude BB Systems of the day by means of his friends and I could not understand that appeal of it, even if he could talk to people from other parts of the world through the computer, it was so black&white and in text-mode screens. It was when I began playing my first games in such BB Systems that I resumed my learning of computers and could program things again in a variation of BASIC called PPE (of the PCBoard BBS). When I migrated to the Internet I got into mIRC Script and Delphi for instance. Throughout the early days of my programming experience, I remember having problem twice, once with programs that seemed to manage pointers (even in PPE) and once in a Perl program which exposed me for the first time to what I would learn later on were Regular Expressions, and it left a scar in me that got cured just very later on when I got to them in Ruby. When I found RegExps in Perl I was like: "If I cannot understand what Perl is doing to a simple string, I cannot understand it at all." Nobody to teach me what it was all about, even my other programmer friend who later on got with Flash/ASP/C# just fine.

In the recent years I had been avoiding getting to terms with C and C++ even if I did have a little of experience with both of them, but I never really created programs with them. Why? Because of Java, .NET, and even I could do with Delphi at some point. ;-)

And nowadays I am not really sorry for not having learned C and C++ more deeply as the chances for using them have gotten a lot rarer nowadays, unless I really cared about getting a C or C++ job, and I do not.

With Web browsers providing us with User Interfaces and Javascript and higher level languages for the server-side stuff, one can do C and C++ either as a hard core lone-wolf or in a team somewhere, and said teams would rather have people in the team who were more like themselves, and not brain-damaged BASIC programmers.

This is not so much about self-esteem, but rather about the dramatic changes brought about changing technologies.

So, does that mean that I did not appreciate coming across more formal ideas in my informal programming steps? Not at all. I have watched some videos on Big-O and so on which were posted on the Internet in the recent years, but they did not get very engaging. I also read a few books and at least one of which exposed some more formal ideas in somewhat practical terms. And even though I have not written any parser deserving praises for their formalities, I have gotten quite experienced in writing them in my own idioms, as when I tried to use something else like ANTLR or some parser generator they always got in the way one way or another. And I am not in the business of creating programming languages, thankfully. (I am damaged, I know.)

I hope you enjoyed reading about another guy's own experience and that you will be certain to enjoy your limitations but that you can keep on learning till very late and always enjoy it even if you will not win many praises in the process.

Cheers.

Lots of fixes for Ruby mode of Text Widget since yesterday

After the initial success of getting enough of Ruby supported to cause an impact, I continued with opening files from the library files present in the Ruby source code repository and every time I found an issue I tried to fix it. The latest series of fixes were demanded from literal regex appearing at the start of a new line as can be seen on this screenshot:

http://img413.imageshack.us/img413/2212/textwidgetrubyregexfixe.png

To make it happen I had the idea of recording the last meaningful token before "turning the line" and using it for the checking of regex alongside the older checking.

Among the changes to supporting this I had to create a variable for holding the last meaningful token type and had to add a new token type called "closing punctuation" for the ")" and "]" punctuations so I could use it for the regex checking.

So long as I can find ways for supporting more of Ruby I am going to do it. Having started with a slim core helps with adding more stuff to it without incurring in too expensive costs for performance.

Update on Ruby support of Text Widget: It's great!

After a lot of hard-working since yesterday, I can say that I am happy with the level of Ruby support I have achieved.

Although I cannot say I have been able to solve all the issues I have found, and I am going to list a few of them below, I can now see almost all of the Ruby library files found in the lib directory of the Ruby source code files just fine, whereas to this day I have not been able to say that with any of the simple editors I have used.

There are editors I have never used like Emacs that can enjoy almost full Ruby support, but Emacs has been favored since the beginning of Ruby. Then there is Vim that can enjoy quite a lot of support as well and it too has got a long history behind it. Then there are commercial editors such as TextMate which may have good Ruby support. When it comes to the IDEs, they can use full Ruby parsers to get their job done and it kind of restricts what an editor can do without having a full parser to back it and also that a full parser can be version specific of the language implementation and it complicates matters when trying to support multiple versions.

With those choices, I am happy with what I have achieved with a custom parser built specifically for spitting out colorized tokens for showing on one editor. If it was not for the flexibility allowed by this, I would have had to settle for less, much less... It is true that I could try and work out to close the 100% completeness gap, but I have to work on other bits of the editor for now.

So, what remains to be supported include:

• After the HereDoc has been declared, the remainder of the line where the initial delimiter appears should continue to be parsed normally without being included in the HereDoc block. This also allows stacking of several HereDocs starting on the same line but with their body implemented in the order of appearance below it. It complicates matter a little as you may see.
• I do not know how Ruby does it, but in at least one library file I saw a multiline string started with the *"* quote and spanned through tens of lines many of which included unescaped *"* characters. While Ruby can find a way out of that mess, I could not so far.

I cannot remember of any other issues right now, but I am going to either edit this post and include them or write a new one.

Here is a screenshot of one feature I achieved in a standardize way:

http://img145.imageshack.us/img145/3795/textwidgetflexopener.png

It shows a string delimited by a flexible matching of delimiters and so long as the delimiters are balanced out throughout the multiline block, they do not need to be escaped. I have a counter that decreases and increases with every delimiter matching to know when it should end. The delimiter can be any non-alphanumeric character, with the added bonus that if it is a matching delimiter, as in "(), {}, <>, []", it enters a matching algorithm as I talked of above and the screenshot shows.

Similar algorithms also worked for execution strings ("%x()"), regex blocks ("%r()"), literal arrays of strings ("%w()"), and even symbols ("%s()")...

What I also have worked on is trying to avoid conflicts between the HereDoc declarations and class modifying with the "<<self" kind of syntax.

I also created code for dedicated handling of method declarations (following "def") and method aliases (following "alias"), so they could include symbols as in "[], []=, **, <<" and perhaps other kinds of supports. As can be seen on the following screenshot:

http://img87.imageshack.us/img87/628/textwidgetdefandalias.png

I also added some custom handling of short symbols that try to be symbols representing potential method names as can be seen in the screenshot with ":/".

All in all it has been a wild experience, if only a little hard to be replicated by other editors and syntax highlighters.

Text Widget gets huge initial Ruby syntax highlighting support, including Here Doc and its variations

Here is the screenshot of getting the Here Doc variations working:

http://img154.imageshack.us/img154/1986/textwidgetrubyheredocs.png

There remains a need to support the Ruby numbers a little more thoroughly as the Ruby mode was started with the one copied from Javascript and Ruby numbers require support for cosmetic underlines as in "1_234_567.890_123". Also Ruby has some String/Char data types started with the "?" character that are not quite supported yet. That is, Ruby requires some more work to call it complete enough, but so far it has progressed well.

Just supporting these Here Doc features of Ruby was a lot unique already. Ruby is one of the very few programming languages that allow such flexibility that an ordinary editor that has to support many programming languages might have a hard time trying to support enough of Ruby to make most Ruby programmers happy.

Little nuisances crop up with every Ruby feature that goes unsupported and while we can adapt to it, as I have many times before, it feels all so great when things are better supported. For example, Symbols create with :'here is a symbol' or identifiers ending in "?" or "!" as in empty?.

What makes Ruby to have a soul is also what makes it all the more unique and hard to support.

While I do not finish supporting Ruby, here is another screenshot:

http://img200.imageshack.us/img200/4890/textwidgetruby.png

To read more about the Ruby features that need supporting, see this awesome summary:

• http://www.zenspider.com/Languages/Ruby/QuickRef.html

And I still am all the more thankful for the Canvas features the newer browser versions have been making all the more complete and available. It is a great feature to which we can delegate much of the drawing responsibility so we can focus on higher level features! See:

• http://www.whatwg.org/specs/web-apps/current-work/multipage/the-canvas-element.html#the-canvas-element

I have already added support for Javascript and Python. With Ruby added, most powerful scripting languages will have been supported and hopefully once the editor is complete, programming will be even more enjoyable.

Javascript is pretty cool and working at such a higher level and being easy to run, makes programming tough tasks if not easy, at least bearable. To see where Javascript can go, just see where PHP went -- taking in consideration their market differences for sure. While PHP has not been perfect, people have created all kinds of libraries for it with the tools they have been given. Javascript is going to get a similar treatment.

Did not take long to support Python decorators in the syntax highlighting of Text Widget

Here they are in their full glory, in a piece of code of the Python backend of the Bespin open source project:

http://img5.imageshack.us/img5/9832/textwidgetpythondecorat.png

I guess I am done with supporting Python for now? :-) Enough features in? :-)

Edit: Oops. Just spotted a bug caused by my code for supporting prefixed strings. It was caused by my mishandling of the "u" and "r" characters which caused them to be missed by the code responsible for determining the identifiers. So words such as "return" were being missed, but I have now updated the code and all is well. I changed the screenshot just to make it hilite the "return" word and be more complete as it should! ;-)

Python prefixed strings are supported on Text Widget as well

I had to refactor some of the string code to avoid even more repetition just to support the prefixed strings. My first tries at implementing it were over-the-top but the end-result has been good:

http://img95.imageshack.us/img95/7254/textwidgetpythonprefixs.png

With that support in place, just now I remember that there is still some kind of annotation syntax that Python allows which starts with the "@" character that I have yet to support. Perhaps I am going to give it a shot after thinking a little more about it.

Python syntax highlighting support arrives for Text Widget

There remains the issue of having to add support for Python's prefixing of strings with variations of the letters "r" and "u", but I just had some fun supporting Python's numbers:

http://img221.imageshack.us/img221/3986/textwidgetpython.png

Those include long integers and its variations and the incredible imaginary numbers.

The toughest was getting the floats that can be created without the initial number ".123" or without the final number "123." because the "." is normally used for referencing properties of objects and by playing with it like that, the "." gets overloaded and can be a little disconcerting getting it to work. Luckily I was able to build on all the work created for supporting the Javascript mode which then got copied to Python and modified for Python's syntax.

Triple quoted strings are already supported as well.

Text Widget scales to tens of thousands of lines of source code

I have just tested loading the large ChangeLog file present in Ruby's repository and even though it's a little outdated in terms of keeping up with Ruby's trunk developments, it enjoys over 66,000 lines and Text Widget tried to syntax highlight it for Javascript and it finished in low seconds. Here's the screenshot:

http://img189.imageshack.us/img189/2647/textwidgetscales.png

I tried to make the Javascript token recognizer fast enough by having it self-contained in one function for returning the tokens so it really relied on just itself and collections of "if" and "switch" for determining the token. Before that I was almost spoiling it by using Javascript objects as Hash tables for some known tokens or token snippets but ended up deleting all of that code and more after replacing it with lower level constructs.

After the Javascript mode was taking shape, I began wondering about why Firefox still felt much too slower when compared with Google Chrome. With the Firebug profiler I found out that the culprit was inside the "calculate_lines" method, so I had to try to optimize it somewhat.

I created a method for measuring the elapsed time of a function passed to it. So I started using it in certain points to see what was going on. This method is more of a companion for the profiler so the profiler broadly tells where the problem is and we can create some embedded tests for measuring some operation or just measuring something if it takes long enough.

I was able to spot a little problem in my assumption that creating a bound function call was faster than just calling it on the object it belonged to because visually I could save the prefixing of the object. That is, instead of calling something like "ctx.measureText(s).width", I could call it as "mt(s).width". Visually appealing, but I found out that I lost some significant time with it on Google Chrome (atlhough when it gets to milliseconds it can be dubious, so the quantity of calls counts) and on Firefox the difference was lesser. The problem with Firefox is that it seems -- at least in my informal tests -- to have a much slower "ctx.measureText" call than the one found in Google Chrome.

So I started optimizing things a little here and there. I was able to avoid calling the "calculate_lines" twice when the line numbering width changed and I did not have wrapping of long lines enabled. Before I started optimizing it, calling "calculate_lines" at least once for some file being highlighted was taking over 6 seconds. Calling it twice meant taking over 13 seconds. If some of the delaying was not taking place for files as long as a few hundreds of lines, perhaps I would not have noticed it. Google Chrome being so much faster can help with hiding such lacking of efficiency which can be more easily spotted when using Firefox.

Firefox by being slower helps with keeping us honest. ;-)

So, optimized a bit, got it down to less than 6 seconds. "Wait a minute", I thought. "There is this 'ctx.font = block.font || font;' being called for every block pass and blocks can be as small as no character or 1 character depending on the construction of the line". I had already protected against setting it redundantly in the 'draw_content' method. "What if I protected against setting it twice for the same font here as well?"

Voila.

Found the real culprit. Got the timing down to less than 1 second (though I have not cared to determine how long after I got it to the 'instantaneous' feel).

The result has been great after all of that. I can load most relatively small files (less than 1 thousand lines) instantaneously with Firefox and with Google Chrome it is even better. Firefox helps us with improving our code, and Google Chrome enjoys it too! ;-)

Fold Java Swing

I had thought about creating a snarky post with the title of "Fold Java Swing" given what these newer versions of browsers have been making available. Or that these newer browsers should have been dubbed "The Java Swing Folders." Anyways...

For example, if you wanted to create a fast and basic text editor and you used just one font size for the whole document you could manage to double or triple the performance found in mine given I have to make these calls to "ctx.font" and have to do other flexible measuring just so a block of text can have its own font name and size.

And when you need that extra performance for your application, you could perhaps find it in some WebKit supporting browser or in some of the newer Javascript implementations that even compile code to machine code and help with speeding "property calls" way up.

And with Firefox providing you with known tools and a need for speed...

I am looking forward to trying the latest Firefox RC with this and more. The Web OS has arrived and its UI is browser based!

Cheers! :-)