Reading GPS data from your Sprint Mobile Broadband USB Modem

Courtesy of the best startup in the world, I just got a Sprint Mobile USB modem (Sierra Wireless 598U). Of course, once you get a new toy, you have to try it out, so on the way back home, I plugged it in and experienced the novelty of surfing the web while on a moving train. One of the options intrigued me -- it said "click to start NMEA."

What's NMEA, I wondered? Turns out it's a data specification for GPS, sonar, and other devices defined by the National Marine Electronics Association. Even better, it turns out you can actually read data directly from this device from the terminal window:

[vincentc@local ~]$ cat /dev/cu.sierra05 
$GPGSV,3,1,10,03,42,254,29,06,45,240,28,09,18,060,24,14,65,180,26*77
$GPGSV,3,2,10,18,38,057,35,21,29,119,32,22,68,014,29,26,73,097,36*74
$GPGSV,3,3,10,27,14,049,18,19,39,293,*76
 $GPGGA,030642.4,3725.433133,N,12207.936951,W,1,09,1.3,-13.2,M,,,,*0D
$GPVTG,,T,,M,0.0,N,0.0,K,N*2C
$GPRMC,030642.4,A,3725.433133,N,12207.936951,W,0.0,,241109,,,A*5B
$GPGSA,A,3,03,06,09,14,18,21,22,26,27,,,,2.1,1.3,1.6*36
 $PSTIS,*61
....

Each of the lines begins with a code whose meaning can be looked up here. Pretty neat! Seems like it wouldn't be too hard to write a script to parse all of the information. For instance, the two most interesting ones are:

 - $GPGSV tells me which GPS satellites are currently in view

 - $GPGGA tells my current location (3725.433133,N,12207.936951,W or 37d 25.433133' N latitude, 12d 207.936951' W longitude), along with my current altitude

Already pretty interesting and rich data --- not sure exactly what this could be used for, but it's definitely interesting stuff ... and I imagine there could be tons of interesting applications to be made around this data.

Loading mentions Retweet
Filed under  //  coding  
Comments (2)
Posted 3 days ago

Use ngrep to inspect HTTP headers

Debugging third-party APIs is painful and tedious. Often times, you'll have to inspect all of the parameters passed to the third-party, make sure they're all there, not malformed, located in the appropriate place, etc.. You could monkey-patch your code to spit these parameters out, but this is also painful. Not only do you have to write more code, but often times the parameters are being modified by some other piece of code after you've logged.

What would be nice is a way to spit out the headers on their way out the door over the wire. That way, you're can see what the third-party sees. Found a nice tool called ngrep that does exactly that. It compiled with no problems on Mac OS X in 10 seconds and started running beautifully:

 

vincentc@local ~]$ sudo ngrep -Wbyline -d en1 port 80
interface: en1 (192.168.1.0/255.255.255.0)
filter: (ip) and ( port 80 )
#
T 192.168.1.3:58716 -> 74.125.19.17:80 [A]
 POST /mail/channel/bind?VER=6&it=45250&at=xn3j34m2pyx=j12w3b-8lzkxt&t=1 HTTP/1.1.
Host: mail.google.com.
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5.
 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8.
Accept-Language: en-us,en;q=0.5.
Accept-Encoding: gzip,deflate.
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7.
Keep-Alive: 300.
Connection: keep-alive.
 Content-Type: application/x-www-form-urlencoded; charset=UTF-8.
Referer: htp://mail .google.com/mail/?ui=2&view=js&name=js&ver=suByI-6Q4II.en.&am=!6zf2FVl6CnB3fth99j1pWy.
 Content-Length: 46.
Cookie: GMAIL_LF=1; S=gmail=fM4yyZY0467bXi71KiVpNQ:gmproxy=eRlqhEcR-oT419BlbGUa9g; GX=DQPOGrd680ZqSK5OXwAl13tBJUp_K79Y4Z9eSknff5gzoKH_gIBlQ9lKXRyx5nPHkqziJso6k_QNM6kC5Sy-xQw0mjV8aWYpr-VRObldXWZgqP2GjjQJBwY9UgMw; GMAIL_AT=xn3j34m2pycti4gjvoyvs2tyyxs0hn; gmailchat= PREF=ID=bbd521a14581b7049:LD=en:NR=10:TM=1242260735:LM=1257471614:GM=1:S=GV8TdngB2-ZjTbi8; SID=DAADFz2eJZ3gDFlQHZE_xjVZk59zWPUG_DNwKt1t6
 ##

Loading mentions Retweet
Filed under  //  coding  
Comments (0)
Posted 18 days ago

Simple perl script for downloading entire books from myilibrary.com

Wow. People really want to download books from myilibrary.com. They restrict downloading, even to students who have legitimate access to their books through their universities. I was somewhat annoyed by this, so I wrote up a quick 5 minute script that would download entire books from Myilibrary.com and posted it out of pure frustration with their stupid policy. Since then, I've gotten a lot of people coming to that post through google.

I've had a few requests for this script so I'm posting it up here. Remember, this script is an old script (I haven't used it for awhile) so I don't know if it will still work. Second, this script will only allow to automate the download process to books you already have legitimate access to (usually through your school or university). Third, you can only download about a hundred pages a day, so you'll have to split your downloading of long books over several days. Fourth, I have no interest in maintaining this script or answering emails about how it works-- I needed it to download two books, and once that was done, I haven't touched the script in several months. If you need more details, check the notes I made that will give a reasonably tech-savvy person everything he needs to know.

[UPDATE: I've been asked to remover this content by a DMCA Complaint initiated by Ingram DV LLC. However, that's not to say any reasonably tech-savvy person couldn't figure it out.]

Loading mentions Retweet
Filed under  //  coding   science  
Comments (0)
Posted 3 months ago

Ruby on Rails Implementation of Microsoft Live Services Delegated Auth

I searched awhile without finding an easy-to-understand example for implementing Delegated Authentication for Microsoft Live Services. I guess the community of people implementing Microsoft Live Services in Ruby is still pretty small. Also, the opaqueness of the documentation made it clear that Microsoft had no interest in making their API's accessible.

 Anyway, I pieced this gist together from MSFT's own example code and documentation. Their examples were long-winded and painful to get through -- this gist just shows the essentials for generating a valid authentication URL. Hopefully others implementing in RoR will find this useful! In the near future, I'll add methods to handle the ConsentToken that is returned after your user authenticates.

A few caveats:

 - This assumes a pretty standard RoR development environment. However, you must require openssl, cgi, and base64. More importantly, your OpenSSL module *MUST* include SHA256 as an installed algorithm.

 - Annoyingly enough, my ruby install on my Mac *DOES NOT* include SHA256 in the OpenSSL module. If this is true for you, then you won't be able to run this code off-the-shelf on your local development machine. Since most developers have access to a Linux box (Ubuntu's ruby install has SHA256), my work around was to extract the code for the 'get_app_verifier' method and put it into a script that would execute on the remote box. Then, if RAILS_ENV == development, I'd just remotely execute that script using SSH and capture the output.

Loading mentions Retweet
Filed under  //  coding  
Comment (1)
Posted 3 months ago

Thinking about using Facebook Connect on your site? Great! Step one: Abandon your will to live.

Facebook Connect, I know you're reading this, and we need to talk. It'll only take a second. It's just you and me here, ok? Just us boys. Tell me what's going on. You don't complete your documentation, your code is full of serious bugs, you don't pay attention to your bugzilla...

Wow. Sometimes you read something that sums up your sentiments perfectly. Captures your internal zeitgeist in a way that you wish you could, if only you were a little more eloquent.

This was it. Coding with the Facebook platform has allowed Posterous to do some really neat things. But there's also a dark side and with the sweet comes the bitter: developing for FB requires patience. A loooot of patience.

Thanks to Garry for pointing this out!

Loading mentions Retweet
Filed under  //  coding  
Comments (0)
Posted 9 months ago

Posterous Continues to Simplify Blogging with Facebook Support

-Posterous Logo-Posterous launched as a simplified blogging process for users that want to stream a wide array of media to the web. Simplicity is key to its design as well as its submission process, lowering the barriers to entry by enabling users to update their Posterous blog by emailing text or media to the Posterous service.

The blogging platform is extending this simplistic view of sharing content by integrating with Facebook Connect. You can now update your Facebook wall by emailing content to Posterous. You can also post Facebook photos directly from Posterous, which will appear on both your Posterous blog as well as your Facebook Photo Galleries. Posting more than 5 photos will create a separate album with the email subject being the album title. Video support for a similar feature is currently being developed by Posterous.

Facebook connect for Posterous is launched! Major props to Garry as we both sat down to debug for IE 6/7 (in all seriousness, probably the worst browsers in the world) , chased down the last few bugs, and even found one that threatened the security of the free world.

Not to mention free beer at 2am ...

Loading mentions Retweet
Filed under  //  coding  
Comment (1)
Posted 9 months ago

Updating Cookies for Facebook Connect

One of the amazing things about Facebook Connect is that it allows developers to incorporate social-content from Facebook into their own sites, break the user free from the confines of the facebook.com site, and offer a more 'seamless' experience from the end-user's perspective.

For those that don't know, the magic of Facebook Connect is largely driven using cross-domain communication within a browser session to pass information back and forth between Facebook and the end-user. State information like the user's login credentials and session information are stored within the user's cookies and a large library of javascript code mediates the transfer of information between facebook, the user's browser session, and the user's cookie stores.

One of the primary challenges facing engineers who use web develpment frameworks like Ruby on Rails is that the heavy reliance on browser-based javascript and cookies is that it breaks the Model-View-Controller (MVC) architecture, leading to all sorts of development headaches. Since the session keys that the server requires to make queries to Facebook's API are stored in cookies and not passed to the controller in a HTTP request, this can lead to strange situations where the session state stored on the server-side does not match the session state stored in the cookies.

The standard solution seems to be simply to force the user to reload the page, which will refresh the cookies, which the server-side controller code can then grab on the next reload. However, this often leads to a poor end-user experiences where the user is forced to reload pages several times in order to syncrhonize the information stored in cookies with the information stored in the server. Even worse, the web site may simply cease to work in certain cases (e.g., if the user logs out of facebook while logged into the current site). For an interesting discussion on this, I found this nice post.

In this post, I'll be discussing an easy way to get around this problem for a simple case that popped up during development of a feature here at Posterous: prompting the user for the "offline_access" permission. I'll run through the situation, the naive solution using page reloads, and the "more elegant" solution (no doubt there are more elegant solutions, but I found this works fine).

The situation

This seems to be a very common scenario. Say your application would like to push content to a user's profile without the user being actively engaged in your site. To do so, the user needs grant your site the "offline_access" permission. This changes the session key from a temporary key (valid for a short amount of time) to a persistent key, which your site will need to capture and store in a database. I'm assuming that you're using Ruby on Rails using the Facebooker gem (though the solution will be more or less general for any MVC architecture).

Naive Solution

The problem with the simple scenario outlined above is that the session keys are stored as cookies which are not passed to your site until the next reload. The native solution would be to have to user reload several times:

1) User visits the site and is presented with a "grant offline access" link. Facebook sends a temporary session_key, which is stored in the user's cookies.

2) User clicks on link and grants the offline_access permission via the pop-up window. Cookies ARE NOT updated at this point. Page is refreshed by some javascript.

3) Server-side controller code grabs a copy of the session_key stored in cookies, which is still temporary, since the javascript code that lives in the HTML hasn't had a chance to update the cookies.

4) Server renders the view, which includes the javascript that refreshes the cookies and stores a permanent session_key. User clicks another link to reload the page.

5) Server-side controller code grabs a copy of the correct, persistent session_key and stores it in the database.


My Solution

1) User visits page, clicks on a 'grant access link'. User automatically redirected another page (call it refresh_cookies) which refreshes cookies automatically, which then sends the user automatically to the original page.

2) Controller code grabs the updated session_key on the next page.

The key to this code is using a bit of Facebook's javascript to ensure that the user is redirected only after cookies have been refreshed. Without this, we can't be sure that the cookies have been updated on the next view. The code for the rails view that accomplishes this is:

<script src="http://static.ak.connect.facebook.com/js/api_lib/v0.4/FeatureLoader.js.php" type="text/javascript"></script>

<script type="text/javascript">

window.onload = function() { 
    FB_RequireFeatures(["XFBML"], function() { 
        FB.Facebook.init('YOURAPIKEY','/xd_receiver.html'); 

        FB.XFBML.Host.get_areElementsReady().waitUntilReady(function() { 
            //alert('All XFBML elements loaded'); 
            window.location.href='DESTINATION';
        }); 
    }); 

}; 
</script>

<h4>Updating Facebook settings for <%= fb_name(@fb_user, {:useyou => "false"}) %> <%=image_tag('loading.gif')%></h4>

By sticking a bit of FBML (rendered using the "fb_name" helper method in Facebooker) into the view and waiting for it to render, you've ensured that the cookies stored in the browser are the most recent ones, which will be captured and saved by the controller after the redirect. The 'loading.gif' is simply a dynamic image that will hopefully prevent your user from thinking nothing is happening. In practice, the simple XFBML on this page takes about 1-2 seconds to load -- a long time, but necessary to ensure that we have correct session_keys.

Thanks for reading -- see you guys on facebooker-talk!

Loading mentions Retweet
Filed under  //  coding  
Comments (2)
Posted 9 months ago

My first bug report

One of the awesome things about open source software is that when you
have tons of eyes looking at everything, your code becomes much more
robust. Even though I'm not super-hardcore-open-source dude, I can
definitely appreciate finding little bugs and contributing in my own
way. It's sort of karmic -- since each bug discovery probably
represents hours of futile-banging-your-head-on-desk time of some poor
developer, other developers have probably collectively saved thousands
of hours of debugging time themselves.
 
So I was very happy to discover my own little bug in the Facebooker
ruby plugin. Within 45 minutes of my post to the facebooker-talk email
list, the fix had been pushed to GitHub, and balance was restored to
the universe. Unfortunately for me, discovering this bug entailed
plenty of futile head banging and wondering "where the hell did it all
go wrong!?!?!" Of course, in the end, the bug came down to one line of
code and the fix was super simple.
 
So here it is, buried deep in the Facebooker codebase, my little contribution:
 

 
return unless Time.at(parsed['expires'].to_f) > Time.now || 
(parsed['expires'] == "0") 

 
;)

Loading mentions Retweet
Filed under  //  coding  
Comments (0)
Posted 10 months ago

Facebook Connect ... Ugh!

I've been poking around and experimenting with Facebook Connect in the
past few days and all I can say is ... ugh. This is no complaint about
the platform itself --- the idea is very nice and I'm know it's
possible to create very nice apps using FB Connect. My main problem
with FB connect is that using it means scattering lots of ugly code
throughout your application -- very messy and very dirty!
 
For instance, take a look at this crazy bit of code. All it does is
look up a user's current friends:
 

 
 
<script src="http://static.ak.connect.facebook.com/js/api_lib/v0.4/FeatureLoader.js.php"
type="text/javascript"> </script> 
<script type="text/javascript">
//<![CDATA[
var api_key = 'YOUR API KEY';
var channel_path = 'Relative channel path here';
FB_RequireFeatures(["Api"], function() {
  // Create an ApiClient object, passing app's API key and
  // a site relative URL to xd_receiver.htm
  FB.Facebook.init(api_key, channel_path);

  var api = FB.Facebook.apiClient;

  // require user to login
  api.requireLogin(function(exception) {
   Debug.dump("Current user id is " + api.get_session().uid);
   // Get friends list
   api.friends_get(null, function(result) {
     Debug.dump(result, 'friendsResult from non-batch execution ');
   });
  });
});
//]]>
</script> 
 

 
Maybe I've drunk the Ruby on Rails kool-aid way too much for my own
good, but to me, this is the nastiest piece of code. For one, it
explicitly breaks the Model-View-Controller way of doing things and
shoehorns all of the programming into the HTML view. Not to mention
that I think intermingling code + html is the worst possible way to
code. It's messy to read, hard to debug, and a pain to work with.
 
Now, I know that RoR doesn't avoid this completely -- you still have
to write .erb files that intermingle code and markup. But the idea ---
the brilliant idea, I think --- is that it moves much of the heavy
lifting of querying databases, programming logic, etc.. away from the
HTML and compartmentalizes these things so that it's much easier to
work with. Now it seems we're taking a step back.

Loading mentions Retweet
Filed under  //  coding  
Comment (1)
Posted 11 months ago

Rubular for regexps

Regexps are the bane of my existence. I write a lot of them for
scanning output logs from my simulation runs, general search/replace,
and other miscellaneous crap. I've always found writing and testing
regexps to be annoyingly difficult, especially trying to think of edge
cases that break your code.
 
Having an interactive shell (like irb) takes a bit of the sting out of
prototyping and testing regexp code, but I stumbled upon this site
yesterday that blows it out of the water. It's a site where you put in
some text and then start typing a regexp pattern. It dynamically (in
real time!) highlights which parts of your text match the regexp
you're typing. If you forget any regexp syntax, there's a reference
box that gives all of the commonly used patterns right at the bottom.
Genius!
 
Simple, cute, and it does the job.
 
Rubular, for regexps: http://www.rubular.com/

Loading mentions Retweet
Filed under  //  coding  
Comments (0)
Posted 11 months ago