Another GSoC and a new blog

I've been selected for Google Summer of Code again! I'll be working with the Instantbird team this time as well, to implement indexed chat history and infinite conversation scrollback.

I'll be blogging my progress this time to http://blog.nhnt11.com. I've imported this blog there as well for archival purposes. A more detailed post about the project will follow on that site.

GSoC '13 is over!

Well, Summer of Code 2013 is officially over. My project has pretty much met what was in my proposal - the stats and ranking patch is r+ (though awaiting Florian's approval) and should land soon.

There are a few leftover bugs that I'll be working on when I have time (I've been pretty busy with my coursework lately). Many of them are small (and possibly quick to fix): bugs 2046, 2048, 2069, 2081, 2116, 2076, 2145, 2146, 2184, 2185.

Here's a broad overview of my work over the summer:

• Modified the tabbrowser to accept generic tabs (bug 426).
• As an example usage of the new tabbrowser changes, made it possible to open about:* pages in a tab (bug 2002).
• Wrote the initial new conversation tab, with a list of contacts and a filter box (bug 2015).
• Added support for existing conversations in the new conversation tab and moved the management of possible conversation data to a new ConversationStatsService (bug 2055).
• Wrote an API to request protocol plugins for chat room lists, and implemented it for JS-IRC. Made the stats service use this API to display IRC channels in the new conversation tab (bug 2066).
• Experimented with IndexedDb to store conversation stats. Dropped it in favor of a JSON cache until we migrate to a more sophisticated database from which we can query possible conversations on the fly (and possibly incorporate log indexing?). Wrote code to store these conversation statistics and use them to sort possible conversations. Adapted Florian's log sweeper code to parse JSON logs for stats (bug 2143).

I've learned a lot over the summer. That includes technical stuff - JavaScript, XUL and the Mozilla platform, XPCOM, etc. - as well as a much deeper insight into open source development. I'd contributed to open source before GSoC, but mostly just small patches. This summer I was actively involved in the development process, and got to see how much care and attention is paid to detail when making even small changes. I had to consider things I would've ignored before - localization, accessibility, edge cases, UI profiling, etc. Every UX decision was preceded by lengthy brainstorming and debating sessions on IRC. It wasn't uncommon to spend 10 minutes (sometimes more) figuring out the best name for a variable!

A huge thanks to my mentor Benedikt Pfeifer, and the Instantbird team for all their help over the past three to four months, they've been a pleasure to work with! Be on the lookout for Instantbird 1.5, it's going to be pretty awesome (:P).

PS
You can download a change log of my project from here, or if you have the source, you can generate one yourself using
$hg export `hg log -b default -u nhnt11 | grep changeset | cut -f3 -d ":"`

Progress report: September 6th

This post has been long overdue. Here's the major progress over the last couple weeks:

• Bug 2066 is merged! Awesometab now queries accounts for chat room lists and displays them (only implemented for IRC, you'll have to flip the chat.irc.allowList preference to true in about:config).
• Bug 2143 - Ranking and stats are working and have had a couple of review cycles. We decided that efficient and intelligent searching/filtering is a requirement and also that the log sweeping code could be reused for log indexing. Based on all this the IndexedDB has been dropped from this bug in favor of a simple JSON cache for now.

First thing I'll be doing next is optimizing the chat room code from bug 2066 to eliminate UI lag while receiving chat rooms. After that, the priority will be landing bug 2143. Things will likely get slow after that while I have my first test cycle, but once that's over (or possibly before, if 2143 lands quickly) I'll be looking at optimization and better searching (implement a proper database, indexing, etc).

Cheers!

Weekly report: August 21st 2013

During the last 7 days, I didn't quite get around to everything I wanted to do (which wasn't helped by 4 of the days being holidays) but there was progress.

• Bug 2066 (chat rooms and IRC's LIST) is finally r+.
• I experimented more with frecency scores ("frecency" is inspired by Firefox's Places Frecency algorithm)
• After a bit of experimenting, I finally decided to stop messing around and get some actual data to play with from logs.
• Using Florian's WIP from bug 1584, I put together an addon that goes through logs and obtains information on recency (date of most recent log file) and frequency (number of messages exchanged, with separate counters for incoming and outgoing)
• Playing with the algorithm for a while, I finally ended up with the following:
• Score = Total message count * frequency multiplier * recency multiplier
• Here, frequency multiplier is the ratio of outgoing to incoming message counts, and recency multiplier is identical to what Firefox uses.
• This gave me pretty accurate results, the conversations with the highest scores were definitely what I would most likely want to open if I pressed Command+T :)

Tomorrow I will be working to integrate this into the stats service and use these scores to sort conversations in the newtab.

PS.
I again notice that my work over a week seems to be concentrated over one or two days, during which I spend 10-15 hours each day on a kind of frenzy. Meh.

3-day report: August 9th, 10th and 12th 2013

• Played with ranking - using frequency as well as recency as parameters (took most of my time on Friday)
• Made changes to the way ranks are stored, to better accommodate chats (soon).
• Updated tabbrowser images for retina displays (bug 2103) (Monday night)

Daily report: August 8th 2013

• Rewrote the way conversations are stored in IndexedDB. This took most of my time tonight (pesky bugs!).
• Now, a conversation's rank is stored with the conversation's id as the key.
• I've defined a conversation's id as the string obtained by concatenating its source and _id values.
• For contacts this looks something like "contact2", "contact-5", etc. For chats, it's source + accountId + name - for e.g. "chataccount1#instantbird".
• Added basic ranking. A conversation's rank is the number of times it has been opened. Ranks are updated and stored as conversations are opened.
• Short term to-do list:
• Code cleanup
• Support for chats
• More sophisticated ranking
• Obtain historical data from logs

Weekly report: August 7th 2013

• More modifications to the chat room list patch (bug 2066) - unfortunately this still isn't ready to land yet.
• Brainstormed the ranking system and storage of conversation data.
• Read documentation on IndexedDB and wrote code to store/retrieve contacts in an IndexedDB object store (mostly experimenting with the API and understanding the approach).

Daily report: August 1st 2013

Today I finally discovered what was causing performance issues I was trying to debug for a couple of days. Lag was being caused by tons of warnings being spewed due to bug 2075.

Having fixed this I was finally able to finish a new patch and upload it for review.

Weekly report: August 1st 2013

• Worked further on bug 2066. Hopefully this should be ready soon.
• Submitted patches for follow up bugs: 2068, 2075, 2076, 2078, 2081, 2082.
• Progress has been a bit slow. I need to speed up and get the chat room stuff done so that I can start working on statistics and ranking.

Daily report: July 24th 2013

• Further work on bug 2066 (WIP attached).
• Brainstormed the best way to cover up the awkward pause between clicking a chat in awesometab and the new tab showing up. For now, made the IRC joinChat method create a new conversation and then send the JOIN message to the server.
• Various improvements, see commits on BitBucket.
• Created patches for followup bugs 2051, 2065. New patch on bug 2055 as well, addressing feedback comments.

Daily report: July 23rd 2013

I've mostly been working on adding functionality to get lists of chat rooms for accounts while the patch for bug 2055 awaits review - in particular IRC's LIST. Here's a progress report:

• Added the necessary additions to imIAccount.idl (an interface that represents chat rooms, and a method in prplIAccount that returns a list of rooms).
• Added implementations for these additions in jsProtoHelper.
• Consumers can now ask IRC accounts for LIST data, which will be returned asynchronously via a callback.
• Created an implementation of PossibleConversation - PossibleChat - to use with these chat rooms. These are now filtered and returned in getFilteredConvs, along with contacts. New conversation tabs can use these to create list elements and open conversations.

To-do:

• Proper data management for the chat room lists that stats service receives.
• Performance improvements, which will likely be helped by:
• Add ability to return list data in batches (Freenode for example has way too many channels to return all of them at once).
• UI improvements for MUC items.

After this is done I will get to work on the statistics and ranking part of my project, which will likely be the main focus of the second half of the summer.

Weekly report: July 22nd 2013

• Made a lot of modifications to the patch for bug 2055 to address the feedback I got for the first one.
• There are now multiple instances of PossibleConversation (PossibleContact, ExistingConversation, PossibleChat to come). The contacts are maintained in one list (as of now) and UI convs are injected into it before filtering.
• Did a lot of refactoring in the newtab binding with clearer naming and future requirements in mind.
• Existing conversations (both already in a tab and on hold) are now displayed in newtabs.
• Fixed a few follow up bugs (bugs 2041, 2045, 2047, 2063).

This week's plan is to get started on the ranking system. First goal is to have LIST working from IRC, then storing the data in JSON (as of now, but I'm keeping the IndexedDB option open).

Daily report: July 16th 2013

• Today I added support for conversations on hold in Awesometab.
• This took most of my day. I spent more time than I'd have liked to while figuring out how the blist window manages convs on hold and how to extract the data I need from the [showing-]ui-conversation[-hidden] observer notifications.
• I've attached a patch on bug 2055 and requested feedback on it.
• I still need to make it display open conversations (but not on hold).
• I plan to do this and have a complete patch with comments ready tomorrow.

Weekly report: July 16th, 2013

The past week:

• The buddy list tab is merged
• Quite a few changes were done in order to address more review comments Florian before the merge.
• Lots of followup bugs have been filed (bugs 2041, 2043-9). I have WIPs for most of them, and patches submitted for a few.
• Did research on the account interfaces for adding methods to return MUC lists (and a bit of experimental code - I was able to open IRC channels from awesometab)

To-do next week:

• Display conversations on hold in awesometab.
• Add the new function(s) in the account interface to get MUC lists.
• Implement this in particular for IRC LIST.
• Display IRC channels in awesometab.
• Also while doing this, address followup bugs to bug 2015 (including the ones mentioned above)

Pseudo-weekly Report: July 10th 2013

I call this "pseudo-weekly" because it's been a week since my last report :(. I was afk for a few days at the end of last week (literally didn't open my laptop :( ) so progress has been a bit slow.

• More review iterations on the buddy list in a tab bug (but 2015)
• Created a new interface ("ibIPossibleConversation") that will encapsulate both contacts as well as non-contacts like MUCs and people on IRC who haven't been added to the buddy list.
• The stats service and newtab binding now work with PossibleConversations instead of imIContacts.

Daily report: July 1st-2nd 2013

Monday, July 1st:

• Mainly a bunch of UI tweaks. This took a while. The new fonts are much cleaner looking, and some icon alignment issues are fixed. Also included the aero version of tag.png.
• Fixed a few bugs relating to key events.

Tuesday, July 2nd:

•  Looked at the implementation of the LIST command on IRC to get some idea of what kind of data I would have to process for MUCs in Awesometab.
• Moved all of the data management code from awesometab.xml to a new service (ConvStatsService) that will in the future manage contact statistics and sorting for Awesometabs.
• Learned quite a bit about services and interfaces (particularly about writing IDL files!) on the way.
• The buddy list tab now uses the service and is fully functional.

Daily report: June 28th 2013

• Fixed some bugs related to observer notifications and the updateContact method.
• Updated my Pastebinner addon to allow the user to set a default language for pastes, and also detect diffs (startsWith("diff") ;) ). Nominated it to be public, and aleth approved the request.
• While doing that, found a bug - Services.cmd.unregisterCommand wasn't working. Filed it and aleth (after "half an hour of debugging"!) found the problem that was right under our noses.
• I went through the code and did a sort of self review - fixed some nits, added comments, did some general clean up. Found a couple of minor bugs and fixed them.
• Incorporated Mic's icon and Firefox's tag icon.
• Finally uploaded the new patch for review.

Daily report: June 27th 2013

• Did some more work on handling of DOM elements. Instead of permanently linking a list item with a contact and removing/adding them when resorting/filtering, now the required number of items is maintained in the list and each is refreshed with the correct contact for its index. I know this sentence may not make complete sense, the code in my repo might make it clearer. This was inspired by a suggestion from Mook on IRC.
• Items are now appended as the user scrolls.
• Performance is greatly improved. Very little to no lag while loading contacts, resorting, etc.
• Added Mic's favicon and also started using the tag icon from Firefox.
• Submitted a patch to add a method to the Contacts Service to return an array of all contacts. This is checked in.
• Started maintaining a short-term todo list in the etherpad.

Daily report: June 26th 2013

• The tabbrowser patch is finally checked in, as is a patch adding an /about command to show about:* pages in a new tab!
• Today I worked on rewriting the awesometab buddy list code to better handle DOM elements.
• I've now got it dynamically removing and adding elements as necessary rather than displaying the whole list and filtering with CSS. Code is uploaded to my repo.
• There was a discussion on executeSoon and avoiding multiple calls to a function in a short time on IRC.

I am going to try and write these logs from now on. They may be of some use later on, or just nice to look back at.

Weekly report: June 23rd 2013

This week the tabbrowser patch saw a lot of tweaking (four review iterations iirc) and is now almost checkin-ready. I also worked on bug 2002 - adding a command to display about: pages in a new tab. This was a good demonstration of the new tabbrowser features and might help iron out any remaining bugs once it lands in the nightlies and users try it out.

I also worked on the awesometab buddy list. There were discussions about the UX on IRC and I experimented with a few UX ideas (async loading of contacts, alternating background for list elements, displaying only a few contacts at a time and loading more while scrolling, etc). Many of the to-dos have now been crossed out (see previous post). Next week I hope to report much more progress on the UX and possibly some initial details of the filtering/ranking system that is a major part of my project.

A few things that I want to mention:

• Mic created a cool icon for Awesometab. Thanks!
• I learned more about the working about XBL. I had a few frustrating moments because I did not realize that bindings were applied only after adding the element to the DOM (because they are applied from CSS) until Mook kindly pointed this out on IRC (thanks for saving me a lot of time when nobody else was around!). This same issue forced me to be extra careful while dealing with addition/removal of contacts from the Awesometab buddy list.
• I still haven't received my GSoC welcome package and hope it comes soon :(