10 Years of my Digital Life

 Today I'm revisiting a topic I've been fascinating (obsessed?) with for many years: my laptop battery (e.g. see past blog posts here, here, here, here...)

10 Years of Data

I started semi-regularly keeping track of my laptop battery's health using coconutBattery back in 2009, with my big 15" MacBook Pro. Once I figured out how to automate the process myself with cron, I began in late 2012 keeping a record of my battery status every minute I used my laptop. Here is the complete 10-year record of my laptop(s) battery charge capacity:

the "cost" of keeping battery data for every minute of use over many years is only a few hundred Mb 

Comparing Mac Batteries

For the hardware nerds (incl. me) who are curious how these batteries hold up over many years of daily use, here are all 4 computers' lives overlaid, including the 2016 MacBook Pro (w/ Touch Bar) I am using to write this:

Interesting stat: the standard deviation of the capacity data (about a ~14 day rolling mean) is a scant 0.6%!

My 2009 MacBook Pro really started to decay at the end of its life (I used it a couple more months past this data). The 2012 Air (red points) was a definite anomaly. In fact, after I blogged about this problem, Apple reached out and replaced the computer with a new model - my favorite computer of all time, the 2013 MacBook Air. Amazingly this workhorse is still under daily use, on loan to a student!

I've given the 2016 13" MacBook Pro with TouchBar a lot of grief, as I've had many problems with it. However, the verdict seems to be in: the 2016 MacBook Pro's battery appears to be the best I've had over the past 10 years! After 2.5 years of use, it's still claiming to hold 90% of it's original charge! I wonder...  if we start making batteries that substantially outlive the host computers, will they be up-cycled? Could we start to see removable Mac batteries again? Pure speculation.

Creativity SHOUT OUT: Check out this music production Twitch stream

I have changed...

Despite all the advances in hardware over the past 10 years, I think the data suggests the biggest change is with myself... or at least how I use my laptops. Consider this slightly hard-to-read figure, showing the distribution of charge fraction (how full the battery is) for my past 3 computers. For all computers I tended to keep them charged most of the time (peaks near 1), but for the 2016 MBP I keep it fully charged (i.e. plugged in) almost always... which is not surprising since I have chargers at home and at the office. These are really more portable workstations, and less "laptops" for me now.

I make the conscious effort to not work in the evenings

Here is the now-classic digital fingerprint, first popularized by Stephen Wolfram, of my life: 1 dot for every data point, tracing the time of day versus years of use. This is a classic diagram for examining the "quantified self". Obvious big features are: I sleep at night, I take a break near dinner, etc. You can also see some seasonal variations, especially during the 2nd half of grad school (2013 MBA, purple), where I seem to work later in the winters (probably getting ready for the AAS conference).

There's some big data errors present in the 2016 MBP data (blue), which show up as stripes here (i.e. every minute was recorded). My laptop appears to have occasional insomnia? Strange... (note: I've tried to remove these spurious days from here on). I don't think this figure tells the right story, however.

A very real change seems to be in the overall use of my laptop. Here you can see the total use (in 1-week bins) of my past 3 computers, with a big running mean (orange line). This figure shows that in Aug/Sept of 2015 (i.e. ~3 on the x-axis here) my laptop use dropped by almost half! The reason: I finished grad school, became a postdoc, and ordered a big iMac for my desk.

My daily computer usage has changed too - this is the figure I'm most proud of. Here I show the time of day used for each computer. Despite using the iMac at work during the day, my evening laptop usage has dropped a TON. Reason: I make the conscious effort to not work in the evenings! This may seem silly to many, but for an early-career academic it's a serious choice.

My work day is now a bit shorter, with a slightly later start, and an earlier end. Reason: this is largely driven by my toddler's daycare schedule. Yay!

But, my workday is also more consistent. See how there's no dip for lunch? I'm in lunch meetings or  eating at my desk most days now. Boo.

Most interestingly (to me), when you fold this data over days of the week (0=Monday, etc), you can see a very big change as well. While I always was fairly good at working less on the weekends, now you can see I have almost NO computer usage on Saturday and Sunday. I also seem to work a LOT more on Friday mornings than I used to... but I think this is because my wife and I work from home some Fridays, and so my laptop usage would seem higher (since the iMac is at my office).

Conclusion
There are many metrics one could use to trace their digital lives - social media activity, emails, screen time (basically equivalent to my battery record). Given the explosion of use my iPhone has seen over the past decade, I'm very glad to see Apple (and others) are finally getting on board with providing this data to users, and (maybe) helping them use their devices more mindfully. As I've written before, the "cost" of keeping battery data for every minute of use over many years is only a few hundred Mb (and probably could be 5-10x smaller if I cared to compress it at all), and so I encourage Apple to make the usage archive part of every computer's "permanent record".

As always, all the Python code to do this analysis and make these figures (including the very clumsy parser I wrote for the ugly battery log data) is on my GitHub! Also, the simple cron script I use is also posted, including detailed install instructions, if you want to try this yourself.


You might also like:

  

What's in an astronomer's tech bag?

A traditional tech video that I've always enjoyed is the "what's in my bag?" series. It celebrates technology, travel, lifestyle, design... but also (for me) is a productivity motivator. These videos to me fall under the "work style" genre - showcasing how people spend their time, organize their lives & workflows, set up their work & creative spaces, and - yes - what they carry in their backpacks each day.

So here I am proud to present my take on this classic YouTube video form: What is in an astronomer's tech bag? 2018 ASTRO VLOG edition!



Honestly I don't carry that much stuff, and I rely on only a few computers/gadgets to do my work. Mark my words:

when it comes to productivity & creativity, less is more

Video: HOW TO VLOG

After my AAS 231 daily vlog series, there were lots of questions about how I produced these videos.

Since it may be interesting to some people, I shot a short "how to vlog" video. Check it out!



Of course, the real answer is: steal ideas from smarter people than myself! In particular, I learned a TON from my brother-in-law, whose vlog is here.

How I Cite

A while back this question was posed on Twitter:

Hey astro-twitter, I'm a grad student writing a paper and I need your help with citation stuff! (@astrobetter seems like a good taggable)

— Tom Rice (@tomr_stargazer) June 8, 2015

I offered to "quickly write something up", and in true academic fashion... it took 3 months!

Quick Guide: how I manage citations in LaTeX for writing papers and my thesis

Note: This guide does not tell you which papers to cite... All I can advice is you should spend a long time reading the literature, especially from before you were born (applies to all age groups)

When I turned it in, my PhD thesis totaled 221 pages front-to-back. Like most astronomers I wrote my thesis in LaTeX. Apparently I could have written it in Word, but I cannot fathom how people write long documents in that program given how frequently it crashes on my Mac. Also I have no idea how it would have handled my citations.

My thesis was the culmination of many years of work, and included text from 3 papers I had previously published in various journals - all using LaTeX. One chapter contains material for a soon-to-be-published paper as well. All told, my thesis included citations to 236 unique sources!

Dealing with so many citations was (nearly) trivial because I had invested early on in managing all my references!

Writing the thesis was still a laborious job, of course.

You could simply type all your citations according to the proper style, and format your bibliography manually. What if you forget one? What if it's a preprint that needs updating? I also assume overworked journal editors would hate you for doing this. Instead, use the \cite command, and follow this guide!

tl;dr

Get bibtex citations for every paper, put them in BibDesk to make a nice "library". Every paper or manuscript, just point your bibliography to this one file. The end.


Here's what my workflow looks like:

0. Install the TexShop bundle

This includes TexShop (a very nice text editor for TeX) and BibDesk. It also includes a handy tool called LaTeXiT, which you can use to easily typeset pretty equations to copy/paste as an image in to presentations!

Other software to organize your library is available. I have no experience with and cannot comment on them, except to say: TexShop is free and stable.

Building Your Library

1. Find papers you like

Every paper I see that I might want to cite later I keep a reference to. 99% of the papers I'm reading are indexed in the SAO/NASA ADS. Occasionally I find papers or articles that aren't, but that's ok. At time of writing I have 883 papers store in my library that I can cite.

2. Get the BibTex reference for each paper

This is the magic of the whole system. BibTex is a bit of tex-nology (hah!) that is from the mid 80's, that provides a standard format for citations in LaTeX

If you're using the classic ADS, here is where you click to find the Bibtex entry for every paper:

If you're using the new ADS, here is where you click to find the Bibtex entry for every paper:

Easy!

This is what a BibTex entry from ADS might look like:

Say, that's pretty readable! Almost every academic journal website will also provide BibTex citations for their articles. So does Zenodo. Everybody wins with BibTex!

3. Put the BibTex reference in to BibDesk

The first time you do this, open BibDesk and create a new library (I call mine "references.bib"). Every subsequent time you do this, open that same library file

Copy and Paste the entire BibTex entry directly in to your library in BibDesk. Be sure to save your library. 

Now the paper has been added to your library!

Probably you want to change the default citation key string. BibTex entries from ADS will have obnoxious default strings like "1988PASP..100.1134B", which in your LaTeX paper you'd have to cite as:

\cite{1988PASP..100.1134B}

Yuck. Instead, double click on the paper entry and edit this field in BibDesk. I'd probably change "1988PASP..100.1134B" to be "bessell1988", making the citation in my paper even easier to read:

\cite{bessell1988}

3b. Add the PDF of the Paper for Later Reading

This is the beauty of using BibDesk as the GUI manager for your library. Each paper you save can also have supplemental things attached to it, like a PDF file! Hopefully someday (soon) we'll all be able to have internet access 24/7 anywhere, and that journals will make web-based articles that we actually want to read. That day is not yet here, and PDF is often the only useful format for a paper.

The manual way:
- Add citation to BibDesk (see #3 above)
- Download the PDF
- With the article selected in BibDesk, drag/drop the PDF in to BibDesk.

The auto-magical way:
Jonathan Sick has an awesome widget that will add the BibTex entry AND the paper to your library. Awesome. I use this a ton, and should buy Sick another beer next time I see him.

Now when you want to read the paper itself, just search your library for the paper (by author, year, etc) and then double-click on the PDF. So helpful!

What your BibDesk might start to look like...

3c. Add Bonus Stuff to the Entry!

Just like you can store the PDF attached to a paper, you can store basically anything. If a useful paper makes a long table available as an ASCII file, I will save the file and drop it in BibDesk. This is especially helpful if you've taken the time to digitize or transcribe something from an old paper!

4. Organize Your Library

Honestly, the GUI and organization is kind of clunky in BibDesk. I had to move and resize lots of panels to get it to look nice. There are a TON of features I don't know how to use, from citation formatting to connecting to web-based libraries.

One nice feature: You can create "Static" groups (basically lists), perhaps by topic or author, and drag papers in to them from your main library. I have groups for major separate research ares that I'm reading about. This is very helpful when you start to forget that one obscure paper in some topic you meant to reference....

You can have BibDesk autogenerate a nicer cite key, so that you just hit Command+k and it will reformat it for you. Very handy! Here's my settings to enable my preferred cite key style:

The Format String is: %a1%Y%u0

5. Use the Library

Now in every paper or thesis you write, point to the same file, forever, the end.

When I'm writing a paper, I usually have to include these lines in the top of the file:
\bibliographystyle{apj}
\usepackage{natbib}

This example requires the journal-specific file "apj.bst" to be available. You can either have it in the paper working directory, or someplace TexShop can always see it.

If your bibliography style doesn't know the standard AAS journal abbreviations used by ADS and others, you might need to include near the top of your document a reference to aas_macros.sty:
\usepackage{aas_macros}

When I want to reference a paper from the library I just use the citation key as needed. For example:
\citep{davenport2014b}

This is the last 2 lines of every paper I write:
\bibliography{/Users/james/Dropbox/references.bib}
\end{document}

6. Typeset Your Paper

If you're using TexShop, this is the typical sequence to typeset:
LaTeX (shortcut is CMD + t)
BibTex (shortcut is CMD + b)
LaTeX (shortcut is CMD + t)
LaTeX (shortcut is CMD + t)

The underlying BibTex magic will create the needed .bbl file locally that holds all the references you've cited. If you mistype a key, or try to cite a paper you haven't downloaded, TexShop will warn you in the second typeset step. LaTeX will organize and format the bibliography as needed.

7. Submit Your Paper

To ensure things work smoothly, usually when I submit a paper I put the references directly in to the manuscript file. This makes 1 less file to upload and keep updated, but there are downsides too...

Just delete the line pointing to your "references.bib" file from Step 5, and copy the contents of "your_paper.bbl" in its place. When typesetting now, don't include the BibTex step.

Probably you don't need to do this usually, but if an automated LaTeX system is messing your references up it can help.

Conclusion

By using this system for many years I was able to copy/paste my published papers in to my thesis without worrying about my references. Whenever I start writing a new paper it's trivial find papers to ensure the bibliography will work.

What did I miss? Want to advocate for a different tool or approach? Am I totally wrong? Tweet @ me or leave a comment below!

8. BONUS: Keeping it synced between multiple computers!

I keep my master references.bib file in my Dropbox folder, which makes it easy to always point at the same file on multiple computers... simple enough.

But I also keep the Papers library (e.g. papers, datafiles, etc - see 3c above) in sync! Make sure you point it at Dropbox folder as well:

PAC-MAN: The End of March Mapness

Well we've come to the end of March. I was too busy the past few weeks to keep the high pace of map posts up, but I think we had a few gems. Next year we'll do even better!

For the final post this month, I'm tipping my hat to Google who have created the best map of the month. Starting today, you can play PAC-MANin your browser using Google Maps!

No seriously, try it!

When you go to Google Maps today (and tomorrow, I'd wager), You'll see a PAC-MAN utton. Click that anywhere and you're ready to play on your local streets

This feels a lot like the in-browser version of PAC-MAN that Google featured as a "Doodle" on May 21, 2010 in celebration of PAC-MAN's 30th anniversary. 

Happy April 1st.

iPhone thickness over time

Yesterday Apple announced the new, super-duper thin MacBook. Thinness has been a metric of some obsession for the past 5 years for Apple (and many other companies), both in the notebook and mobile phone product lines. Perhaps it mirrors our society's anxiety about body image...

For me, a thin phone makes carrying the device constantly much more comfortable. (Massive screen size is not a big selling point for me) So I was curious, how has the iPhone shrunk since it's conception in 2007?

The figure above shows a nice steady trend towards thin... Using a linear extrapolation to this data, we can expect that sometime in 2023 Apple will announce an iPhone with a thickness of 0mm. That will be 16 years after the introduction of the device, so I project the product name will be approximately iPhone 17.

You heard it here first, folks!

Radio Maps II: Nielsen BDS Stations

In part I of this series of posts I made this neat looking map of all FM radio station transmitter coverage in the US:

This is basically a population density map of course, but the data is intriguing for many reasons and the subject matter of listening to the radio on road trips is full of nostalgia.

Once I started making this map I immediately knew that I had to do a few studies on this fascinating dataset! My next goal was to start grouping the stations by genre information. As a first example of this, I am looking at the ~1600 stations that the Nielsen company monitors (excluding AM and Canadian stations, so actually closer to 1200 stations). Here I'm matching/joining the FCC's transmission coverage data to the Nielsen BDS stations using their call sign (e.g. KUOW). I then grouped the stations by genre/format, and used the Python Basemap package (very sloppily) to draw the geographies

Here's the first image in the set:

Your first reaction might be: is this just another population density map?

Answer: no. The Nielsen BDS stations listen to targeted music/programming in certain markets. They don't listen to every Adult Contemporary station throughout the US, only selected stations in key markets. How these markets are determined is beyond me, and probably a matter of trade secrets or voodoo, I'd suppose...

So what we have in these maps is more like the geographic regions where each radio format/genre is considered important, or are good predictors of sales/popularity.

All caveats aside, there are some very interesting trends in the geographies vs genre, which largely track racial and socioeconomic distributions.

Here's an album of all the images (direct link here incase the widget isn't working well)



For my next post on the distribution of radio stations in the US, I'm working to compile a larger database of most every station with a known format. So far I'm aggregating tables from wikipedia, but if you have a line on a more complete list drop me a line! Then I'll be able to discuss the actual geography of musical tastes in the US.

The long term goal for this ongoing project is called RadioTrip, where users could get map directions and radio suggestions along the way based on musical taste. If you want to help with this project, let me know or ping me on GitHub!

How Long Between Apple Releases?

As an Apple computer fan I've been a long time reader of Mac Rumors, a website that reports great Mac news and rumors. One great feature is their Buyer's Guide, which tracks the refresh history data (and rumors) to suggest when products are due for an upgrade.

I've been wondering if a) Apple products are actually getting cheaper over time, and b) if Apple is refreshing products faster?

Here's the base cost versus the days since last refresh for a bunch of current Apple products on Mac Rumors

Map of FM Radio Station Towers

Here's a curious map I made.

I was recently driving in the southwest, cruising along long stretches of highway that get no FM radio reception. Usually we need to bring CDs or hook up the iPhone to the car, but we were lucky enough to have a rental with SiriusXM, and it was pretty awesome... but I digress.

As a child my dad told me that FM basically only worked along line-of-sight, and not over very long distances, and that's why we had to listen to The Cars on cassette while driving to the Grand Canyon instead of the radio (I kid, Dad. And also I love The Cars still).

So while I was driving along HWY-380 in New Mexico I started to think about the distribution of radio coverage. To cover most of the country there must be thousands of radio towers! Indeed, there are...  around 27,000 of them in the US alone! Here's a map of their coverage across the country...

(click image for high res)

Battery Life, 6mo Checkup

My new 2013 MacBook Air arrived 6 months ago, after a fascinating discovery that the battery performance on my 2012 Air wasn't keeping up with my previous 2009 MacBook Pro's stats.

This finding was the result of a detailed stats keeping, taking a sample of my battery's status every minute I use my computer. Of course, as soon as I unboxed my new Air I installed the battery-logging script and started passively collecting data once again.

I'll certainly revisit this again in the summer once I have a full year's data to compare with, but with 6 months logged I can already point to some intriguing differences...

Tweets & Readability

As I've mentioned previously on this site, this past summer I had the great fortune to intern with MSR in Redmond, WA. Much of the summer was spent discussing, imagining, and thinking about data and science. Additionally, I spent the last few weeks there writing a short paper on one of the data explorations we undertook with Twitter! (I also made a poster about pataphysics, video games, and pandas... but that's another story)

I found writing a paper in another field delightfully challenging. It's like going back to kindergarten...  you have to learn the language, the structure, the pacing and voice. Most importantly, you have to stumble through their literature, trying to appear competent enough to contribute to the scientific process! (Mostly you try to quack like the right breed of duck without looking like a total fraud!) My mentors at MSR helped in this last step as much as they could.

The publication process in CS is quite different from Astronomy. For example: publishing in conferences instead of journals, two-way anonymous refereeing, low acceptance rates.  I enjoy the sheer number of places you can submit your work. In astronomy we have a fairly small number of respected journals to cite literature from, while CS seems to have endless numbers of specialized conferences on every sub-discipline. Pros/Cons to both models abound.

I submitted my paper to a well regarded conference, but eventually it was not selected (though reviews were quite positive!) Probably I'll make another set of changes and submit it again. In the meanwhile I wanted to give it a stable online home, so I did what any astronomer would do: submit it to the arXiv.

"The Readability of Tweets and their 

Geographic Correlation with Education"

Check it out!

A major part of this project was based on the US Census, which (as ever) was a fascinating data source. Here's a figure not included in the paper, but made from Census data. It shows the relation between median household income and the fraction of college degree holders within a given ZIP code.

Remember kids: correlation != causation.... but stay in school.

The paper outlines how we gathered a large sample of Tweets and measured their Readability (reading ease). Here's a cute figure for tweets with geo-data (lat,lon), grouping in to ZIP code areas and measuring the average readability (high reading ease #'s = simpler sentences). No large scale coherent trend is present, but there does appear to be sub-structure. This is something I'd love to follow up more, using some actual statistical/spatial analysis.

Finally, this is the "money graph" for the paper. Here we've shown the average reading ease score in each ZIP code (actually a ZCTA) compared with the % of college graduates. There is a significant anti-correlation present, which I think is very interesting! More intriguing, we didn't find a strong correlation with median income, nor the high school graduation rate.

Average Readability score as a function of college graduate rate. Lower scores indicate more complex text.

A few things could be the underlying cause of this apparent relation:

1. There are significant demographic differences between ZIP codes with very high #'s of college grads and those without. These higher educated people may use more complex language in their tweets, but this seems too speculative to be convincing to me.
2. The content type of tweets may be different in these higher education ZIP codes. For example, promotion of news/events versus personal status updates. Content-tagging a massive number of tweets is needed to understand the dependence content has on linguistic complexity.

To my knowledge only a handful of (very interesting) studies have investigated linguistic complexity within Twitter, and none I'm aware of in its geographic or regional dependence. The neat thing about Twitter is that it is a (massive!) living data set, and you can repeat these experiments every day.

Just for fun, here are a few neat projects/studies being done with data gathered from or derived from Twitter:

• Hedonometer (global happiness measured from Twitter)
• Geography of Happiness
• Anagramatron

TwainBot: Early Lessons

Today TwainBot hit the 100th tweet in Tom Sawyer. It's been about two weeks since the project was launched and I wanted to jot down a few thoughts/lessons learned so far. Some are obvious, some were expected, some I found subtle or surprising....


1. Uptime is difficult, especially for sloppy side projects in new languges. So far TwainBot has gone down twice and missed tweets (I caught it up manually). Once was due to coding errors (my bad) As every sysadmin or dev knows, you are garunteed to release code with bugs, and probably they will cause your system to crash in ways you didn't expect. TwainBot is very robust because it is very simple, and it saves things to dropbox so I can check on it in real time. Still I forgot to ensure that *every* tweet was correctly binned to 140char, and in a few cases (about a half percent of the tweets) my logic statements screwed up and put a few too many chars/words in. This causes the Twitter API to crap out and return an error (smart), but I didn't catch them in any kind of error trap. A programming 101 mistake (facepalm) that was trivial to correct.

2. Uptime is difficult, especially for the Pi. Of course, the Pi is not a server. Heat is an issue. A book is not a good case for a computer. The first tweet of day #2 TwainBot failed. I was home sleeping in on that Saturday morning. At first I thought I had screwed up my cron job and it wasn't scheduled properly, but I couldn't remotely access TwainBot for some reason. Instead it turned out I had cooked the Pi (hah). It overheated and shutdown (showing a red light on the board only). Cooling is a real issue, even for this mini computer. The book is now open, and so problem solved... for now. From reading about the Pi I expect more performance/uptime issues in the future, including possible HD failure. More redundancy is needed, and I'm updating my scripts to push/pull more from dropbox, allowing me to remotely intervene and push new code as needed as I'm traveling a lot next year.

3. A year is a long time. This is a long endeavor, and that's a commitment of time and brain power to run for me. For the audience there's a constant interaction needed to keep up as well. I failed to appreciate this point at first. Some people will follow and read a few tweets due to novelty/amusement. Some will think "Neat! But I'm already behind, screw this." A serious question remains: is this project interesting to anyone half way through? How could I make it more approachable? Is there a way to catch up (you know... besides reading the book) that I can implement? Twitter followers have been joining and leaving, at nearly a constant rate since day one. Time will tell if that means I'm net losing people due to lack of interest, or if interest can really only be fleeting for TwainBot, which brings me to my next point...

4. What is success? TwainBot is an art project, first and foremost. Still, what do I want at the end of the year? 10k followers? 10 followers? The struggle to maintain an audience I think is partially related to a lack of a clear end goal on my part. A lot of people have asked "why are you doing this?". The answer is because I wanted to create something. I didn't go in to this with a motivation beyond creating something to share with people. My end goal now is to talk to as many people as I can about it, to have conversations about what it means to read a book at a very different pace than normal.

5. Everyone (especially on the internet) is a critic. Some people hate destroying books physically. Some people hate "destroying" the art, by stuffing a classic in to a medium it was never designed for. This goes directly back to #4. I want to have these conversations, and every snarky (or positive) comment on the internet is a tiny conversation people are having about the project. Ultimately that's the goal, that people will think more about the intersection between literature and technology.

...and to read books, because they're awesome.

Laptop Battery, the Aftermath

The summer is coming to a close, and so is my internship at MSR. I haven't had much time for blogging these past 3 months, but the few posts I managed to write fared quite well! This included Airports of the World (21k views) and The De-Evolution of my Laptop Battery (114k views).

The Laptop Battery post quickly became my most viewed article (at least in terms of views on this website). At one point it was the top post on Digg, the front of Slashdot, and near the top of Hackernews. This induced a ridiculous traffic spike...

The little code snipped I posted on github got a number of "forks", and the comments on my blog (and other sources) were awesome, insightful, and usually quite friendly (as far as internet comments go).

The Reddit Effect - II

Today I'll share a couple observations about web traffic. Take from it what you will.

Below are two charts/tables, directly taken from my "dashboard". The first lists my Top 10 Articles, ranked by total numbers of pageviews. This shows a smooth exponential-ish distribution, not too heavy on any single article.

(apropos: I really like the built-in stats tools with Blogger!)

The second chart lists the Top 10 Traffic Sources for this site. The traffic from Reddit is more than double that of all other sources combined! WOW! 

This isn't to say that Reddit is the best place to advertise your work. It can go largely unnoticed if you don't participate in the Reddit community, and getting traction within any social news aggregator is often a subtle game. However, your potential exposure can be much higher than places like Facebook.

These stats also don't account for external exposure. For example, I'd wager more than 865 people read Huffington Post's coverage of my Starbucks post.

My intuition is that I need to diversify my readership sources some, that Reddit doesn't necessarily create a stable base (for a host of reasons). But I'm making this whole blog thing up as I'm going, just trying to do my best, so who's to say what's "best"?

Visualizing the Evolution of Cars

From user 27147 on Flickr, Creative Commons licensed, some rights reserved.

I've always been a big fan of cars. Like with sports, cars are a subject for which people love to quantify things. Horsepower, curb weight, engine displacement, top speed, miles per gallon... all of these attributes are carefully measured, graded, evaluated, and play a factor in the car market. In high school I knew every stat for my '87 D50...

When I looked at cars over the last couple decades, the change in one particular attribute stood out to me: cars are getting big! (both heavier and larger) I'm not the only one noticing this, of course, and cars used to be very large. So I asked myself: How have car sizes, and efficiencies, changed with time?

I started by looking at data for one of my all time favorite brands: BMW

The Reddit Effect

Today I'm going to discuss my thoughts on a silly topic that has gone by many names over the years. These days I see it dubbed the "Reddit Effect", usually meaning the impact on your website from being posted on Reddit. This has also been called the "digg effect", the "twitter effect", the "stumbleupon effect", the "slashdot effect"...  Basically an explosive increase in web traffic to your blog/site that can cripple servers, and has the same (though accidental) result as a DDoS attack.

Of course, bloggers/writers crave this kind of attention. Your home-grown webpage can suddenly get exposed to 100K viewers overnight, if you can navigate the social media/news network waters. Nobody can sign up for your email list, however, if your discount webserver gets fried under peak traffic load. Thus, it's a sensible and practical phenomenon to study.

Password Strength

Here's a fun and simple experiment I recently did...

Some time ago I started using phases to remember passwords. This can work a couple different ways:

1) use the first letter from every word in a sentence.

Example: The quick brown fox jumped over the lazy dog = Tqbfjotld

2) use a whole sentence as a password. 

Example: The quick brown fox jumped over the lazy dog

When I attended SDSU, they required the most obtuse password rules. It was something along the lines of "must have uppercase, lowercase, numbers, and symbols. Cannot use more than 2 of each consecutively. Must be more than 8 characters." Oy...

The Graph that Wasn't There

Yesterday I wrote a post on this blog that, upon reflection, was far more cynical than I ever intended. I have removed it, not to hide or to acquiesce, but because I have always wanted this blog to be about celebrating science and data and visualization.

Here is the surviving piece of that post, which I will leave as a means to start a conversation (and a future blog post):

Visualization Design: It's not about obtuse color theory, or infographics, or artistic style, or minimalist Tufte chart theory, or fancy-ass 3D plotting with the latest/hottest software. It's about effectively communicating your story to other people.

Plots that Changed the World

Be sure to subscribe for updates on this and all my other data analysis projects! 

I spend a lot of my time looking at plots (aka graphs, charts, figures...)  If done carefully, graphs can be informative, artistic, personal, edifying, and even clever/witty. So often they are hasty, convoluted, gritty, and confusing. Frequently you encounter figures that seem to have been made out of spite for the data, or to merely satisfy tradition in the field. 

There are a wealth of opinions on what constitutes a good graph, and the philosophies behind creating them. Since there is commonly little formal training for scientists in the art of visual communication à la graphs, I think there is much still be said and written on the subject. I hope to bring some outside expertise and opinions to If We Assume on this very matter. 

One aspect about graphs that has entranced me is their ability to tell a story. A universal tenet of effective figures is that they must tell the story better than any other medium, better than a text description, a table, or an equation. Otherwise they are redundant. By visualizing the story (actually the data) we make it more human, more digestible. It touches chords, associates with memories, and should affect us like a good parlor trick.

The title of this post is a double entendre. I am presenting five plots/graphs that are famous, and have changed the world in a way. Each figure also represents a plot/story, capturing and quantifying the human saga throughout some piece of history. Most are not attractive enough to merit studying their aesthetics unfortunately.

Laptop Battery Lifetime

Be sure to subscribe for updates on this and all my other data analysis projects!

One of the biggest selling points for new laptops is the promise of amazing battery life. I work constantly on my laptop, and have given up using any other computer on a daily basis. A 7-10 hour battery would be an absolute god send.

To measure battery life you have to run the computer from 100% to 0% under controlled conditions. I don't have time to play lab-rat with my laptop, so the only way to explore this is through the battery capacity stats that the computer uses to predict how much charge you have left.

As mentioned in a previous post, I've been keeping stats on my laptop battery for the entire life of my computer. The program coconutBattery makes it easy to save basic information, and I eventually put it in my startup items so it prompts me very time I restart my computer.

Lifetime Trend

Battery capacity of my MacBook Pro over 3+ years and
730 load cycles. 100% = 4600 mAh. Orange line is 3rd-
order polynomial least-squares fit.

Here we have all the data I've gathered on my battery over the entire 3+ years of ownership. (Here's another blog who has done a similar analysis over a shorter amount of time.) I've tried hard to treat my battery properly, and still get 2.5 hours of use.

I got clever with a cron job & shell script that saved battery data every minute my computer was being used (doesn't run when asleep), shown in blue. This gives us an idea of the uncertainty on the capacity measurement from the battery.

Oversampling

Battery capacity readings taken at 1-mintue intervals for ~11 days.

Here is the 1-minute data, covering about 11 days of laptop use. The battery capacity jumps around by ~5%, but no clear trend of when it jumps. The cron job ran without complaint, and I never noticed any decrease in laptop performance.

Usage History

Computer (black) and email (blue) usage versus time.

Lastly, the 1-minute battery data contains a fascinating byproduct: it keeps a detailed record of when I'm using my computer. Shown in black is a histogram of the number of battery samples taken over the 11 days, in 30 minute bins. Overlaid in blue we see the number of emails sent over a year in 30 min bins, taken from a previous post.

The very early morning battery data is a byproduct of me using my laptop to observe with the Apache Point Observatory 3.5meter telescope during one of the 11 nights. Otherwise, my usage pattern is very similar to that of my emails, but with more emphasis on the evenings (again probably due to observing... and Netflix)

Conclusions

Keeping the battery log for 11 days required about 2.4 MB of space. This could be trimmed down with a more careful script that saved fewer characters, but the point is clear: it is trivial (under 300 MB) to save 1-minute battery data for the entire 3-year life of your computer. If you only took samples every 5 minutes, you could still get very accurate sampling of the battery health, and your general computer usage, at a cost of 50-60 MB of hard drive space. In today's world, this is the same as free.

My computer is out of warranty, I feel like I'm living on borrowed time. Rumors are swirling about possible upgrades to the Apple notebook line soon, and I'm eager to purchase a replacement. When I do, I will surely be implementing my little cron script and recording fascinating computer usage data!