Saturday, May 16, 2015

Review of top 10 online Data Science courses

As more and more of life’s day-to-day work and personal activities are being simplified by Big Data technologies, the need for data scientists has risen remarkably for the past several years. Companies around the world scamper desperately to grab people with data science skills, and are willing to shell out big bucks to keep these data-crazed workers in their payroll. Experts agree that data science is still in its fledgling state, it will become a pervasive force pretty soon. If you want to learn data science and become a data science expert, check out our reviews of the following courses!
1) Harvard Data Science Course
The course is a combination of various data science concepts such as machine learning, visualization, data mining, programming, data munging, etc. You will be using popular scientific Python libraries such as Numpy, Scipy, Scikit-learn, Pandas throughout the course. I suggest you to complete machine learning course on coursera before taking this course, as machine learning concepts such as PCA (dimensionality reduction), k-means and logistic regression are not covered in depth. But remember, you have to invest lot of time to complete this course, especially the home work exercises are very challenging.
If you are good at statistics and programming take this course. 2014 version of Harvard data science course is going on. You can access the lecture videos here.
Prerequisites: Cs50 and Stat 100
Programming Language: Python
Course Length: 4 months
Difficulty: Very high
Taught By:  Hanspeter Pfister and Joe Blizsten
 Reviews by others:

2) Analytics Edge

The course gives a good intro to R and also gives hands on experience with statistical modelling techniques. The course has real world examples of how analytics have been used to significantly improve a business or industry. The workload is high, but the lectures and problem sets are well organized and structured. If you’re interested in learning some practical analytic methods that don’t require a ton of math background to understand, this is the course for you.
Prerequisites: Basic knowledge of mathematics
Programming Language: R, Libre office/Excel
Course Length: 11 weeks
Difficulty: High
Reviews by others:  Course Talk.

3) Machine Learning Course On Coursera

Data science and machine learning are closely related. Apart from machine learning the course shows you how to handle high dimensional data(Pca), introduces to map reduce, bias vs variance, learning curves, etc. The course is taught using Octave( alternative for matlab), there are set of videos that shows you how to use octave. It is better to have some knowledge of calculus before taking this course, so Consider taking MIT multivariable calculus course.
Prerequisites: Basic linear algebra and Calculus
Programming Language: Octave
Course Length: 11 weeks
Difficulty: Low
Taught By:  Andrew Ng
Reviews by others:

4) Data Analyst Nano Degree Udacity

A Nanodegree, provided by Udacity and AT&T , is an online certification that you can earn in 6-12 months (10-20 hours/week) for $200/month. Udacity’s Data Science track teaches R, Python, MongoDB and Hadoop. The courses cover both theory and practice of Data Science, and every course ends with a project that allows you to demonstrate what you learned. The projects can be the start of your portfolio of work to share with others, especially recruiters. The prerequisites are pretty high, you need a variety of skills before taking this course.
Course Length: 12 months(10 hours/ week)
Difficulty: Very high
Taught by: Chen Hang Lee and Miriam Swords Kalk

5) Introduction To Computational Thinking And Data Science

The course provides a brief introduction to plotting, stochastic programs, probability and statistics, random walks, Monte Carlo simulations, modeling data, optimization problems, and clustering. Even if you have little programming experience you can learn a lot from this course. The course serves as a motivation for the beginners in Python and data science.
Prerequisites: Introduction to computer science and python programming
Programming Language: Python
Course Length: 9 weeks(12 hours/week)
Difficulty: Intermediate
Reviews about the course:  Mooctivity

6) Coursera Intro To Data Science Course

The class gives a broad introduction to various concepts of data science. The first programming exercise “Twitter Sentiment Analysis in Python” is challenging, and rest of the assignments requires less time commitment. Professor Bill Howe assumes that you know statistics, Python, and SQL, you really need to know them because the lectures are so poor. Before taking this course, go through standford’s data base course, learn Python programming concepts from code academy, learn basic statistics, and basics of machine learning. Don’t expect this course to introduce you to these concepts. Despite its shortcomings, the course explains a lot about relational databases, Map Reduce and No -sql. The course is not intended for beginners.
Prerequisites: Basics of Python, statistics, basic knowledge of databases.
Taught by: Bill howe
Programming language: Python and R
Length: 3 months
Difficulty: intermediate
Reviews about the course: Course Talk and Quora

7) Johns Hopkinson Data Science Course

R programming, exploratory data analysis and cleaning data modules are really well taught and practical. The statistical inference and regression model structure modules are not well organized; they have too much material for some one new to it. Data scientist tool box module is a waste of time; you can see the reviews of this module here: data science tool box.
Project swirl is a fun way to learn R. Most of the time the professor just reads the slides without adding any additional information. Certain concepts are not clearly explained, so you will spend more time googling and learning those concepts. It is too traditional and too heavy in statistics in particular.
Prerequisites: Working knowledge of mathematics up to algebra and some programming knowledge.
Taught by: Brian Caffo
Programming language: R
Length: 12 months
Difficulty: Intermediate
Reviews about the course: Tech Powered math

8) Foundations Of Data Analysis

The course focuses only on statistics and gives hands on experience with descriptive and inferential statistical concepts in R. If you want to learn R and statistical concepts, then this course is for you. You will be working with a well formatted data sets, so you won’t be learning data munging in this course. At end of this course you will be comfortable using different statistical techniques to solve your own problems about your own data using R.
Prerequisites: None
Taught by: Micheal J. Mahometa
Programming language: R
Length: 13 weeks(3-6 hours/week)
Difficulty: Intermediate
Reviews about the course: Course Talk

9) Data Science In Action

The course is based on the book “Process Mining” written by professor Wil van der Aals. If you are a business professional and don’t have prior programming experience, then this course is for you. The course acts as a classical divide between “business” and “IT”. The course uses many examples using real-life event logs to illustrate the concepts and algorithms. After taking this course, you will be able to run process mining projects and have a good understanding of the Business Process Intelligence field.
Prerequisites: A basic understanding of logic, sets, and statistics (at the undergraduate level).
Taught by: Wil Van Der Aalst
Programming Language/Tools:  ProMDiscoRapid Miner
Length: 13 weeks(3-6 hours/week)
Difficulty: Easy

10) Mining Massive Datasets

The class will introduce you to fundamental algorithms and techniques to deal with Big Data, such as MapReduce, Locality Sensitive Hashing, Page Rank, and algorithms for Large Graphs and Data Streams. It will teach you how to apply these tool-kits to important practical applications, such as Web Search, Recommender Systems and Online Advertising. This course gives a special attention to dimensionality reduction. A book based on this course is available for free. The course expects you to have good knowledge of database and algorithms.
Prerequisites: Basic course on algorithms, data structures and databases
Programming Langauge/Tools: SQL
Length: 7 weeks
Difficulty: Intermediate
Reviews about the course: Quora

Conclusion:

Some of the upcoming courses in data analysis are computational methods for data analysis, data analysis and statistical inference, coding the matrixData camp is a great place to learn R. To learn Python for data analysis, you can check out my post.
So, what other courses  are worth taking if you want to get a good education in data science?
- See more at: http://bigdata-madesimple.com/review-of-top-10-online-data-science-courses/#sthash.fnUC81LT.dpuf

Monday, May 4, 2015

Doing Data Analysis and Data Science in Python with pandas

Ever wonder how you can best analyze data in python? Wondering how you can advance your skills beyond doing basic analysis in tools like MS Excel or SAS? Want to learn how to do the data analysis in python and pandas? Then read this post. 
pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real worlddata analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. It is already well on its way toward this goal.
pandas is well suited for many different kinds of data:
  • Tabular data with heterogeneously-typed columns, as in an SQL table or Excel spreadsheet
  • Ordered and unordered (not necessarily fixed-frequency) time series data.
  • Arbitrary matrix data (homogeneously typed or heterogeneous) with row and column labels
  • Any other form of observational / statistical data sets. The data actually need not be labeled at all to be placed into a pandas data structure
The two primary data structures of pandas, Series (1-dimensional) and DataFrame(2-dimensional), handle the vast majority of typical use cases in finance, statistics, social science, and many areas of engineering. For R users, DataFrame provides everything that R’s data.frame provides and much more. pandas is built on top ofNumPy and is intended to integrate well within a scientific computing environment with many other 3rd party libraries.
Here are just a few of the things that pandas does well:
  • Easy handling of missing data (represented as NaN) in floating point as well as non-floating point data
  • Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects
  • Automatic and explicit data alignment: objects can be explicitly aligned to a set of labels, or the user can simply ignore the labels and let , , etc. automatically align the data for you in computations
  • Powerful, flexible group by functionality to perform split-apply-combine operations on data sets, for both aggregating and transforming data
  • Make it easy to convert ragged, differently-indexed data in other Python and NumPy data structures into DataFrame objects
  • Intelligent label-based slicingfancy indexing, and subsetting of large data sets
  • Intuitive merging and joining data sets
  • Flexible reshaping and pivoting of data sets
  • Hierarchical labeling of axes (possible to have multiple labels per tick)
  • Robust IO tools for loading data from flat files (CSV and delimited), Excel files, databases, and saving / loading data from the ultrafast HDF5 format
  • Time series-specific functionality: date range generation and frequency conversion, moving window statistics, moving window linear regressions, date shifting and lagging, etc.
Many of these principles are here to address the shortcomings frequently experienced using other languages / scientific research environments. For data scientists, working with data is typically divided into multiple stages: munging and cleaning data, analyzing / modeling it, then organizing the results of the analysis into a form suitable for plotting or tabular display. pandas is the ideal tool for all of these tasks.
Some other notes
  • pandas is fast. Many of the low-level algorithmic bits have been extensively tweaked in Cython code. However, as with anything else generalization usually sacrifices performance. So if you focus on one feature for your application you may be able to create a faster specialized tool.
  • pandas is a dependency of statsmodels, making it an important part of the statistical computing ecosystem in Python.
  • pandas has been used extensively in production in financial applications.

Lessons for New pandas Users

  • 01 - Lesson: - Importing libraries - Creating data sets - Creating data frames - Reading from CSV - Exporting to CSV - Finding maximums - Plotting data
  • 02 - Lesson: - Reading from TXT - Exporting to TXT - Selecting top/bottom records - Descriptive statistics - Grouping/sorting data
  • 03 - Lesson: - Creating functions - Reading from EXCEL - Exporting to EXCEL - Outliers - Lambda functions - Slice and dice data
  • 04 - Lesson: - Adding/deleting columns - Index operations
  • 05 - Lesson: - Stack/Unstack/Transpose functions
  • 06 - Lesson: - GroupBy function
  • 07 - Lesson: - Ways to calculate outliers
  • 08 - Lesson: - Read from Microsoft SQL databases
  • 09 - Lesson: - Export to CSV/EXCEL/TXT
  • 10 - Lesson: - Converting between different kinds of formats
  • 11 - Lesson: - Combining data from various sources
<Note> This documentation assumes general familiarity with NumPy. If you haven’t used NumPy much or at all, do invest some time in learning about NumPy first.
For more resources, please visit the main repository.

pandas Cookbook

The goal of this cookbook (by Julia Evans) is to give you some concrete examples for getting started with pandas. These are examples with real-world data, and all the bugs and weirdness that that entails.
Here are links to the v0.1 release. For an up-to-date table of contents, see thepandas-cookbook GitHub repository. To run the examples in this tutorial, you’ll need to clone the GitHub repository and get IPython Notebook running. See How to use this cookbook.
  • A quick tour of the IPython Notebook: Shows off IPython’s awesome tab completion and magic functions.
  • Chapter 1: Reading your data into pandas is pretty much the easiest thing. Even when the encoding is wrong!
  • Chapter 2: It’s not totally obvious how to select data from a pandas dataframe. Here we explain the basics (how to take slices and get columns)
  • Chapter 3: Here we get into serious slicing and dicing and learn how to filter dataframes in complicated ways, really fast.
  • Chapter 4: Groupby/aggregate is seriously my favorite thing about pandas and I use it all the time. You should probably read this.
  • Chapter 5: Here you get to find out if it’s cold in Montreal in the winter (spoiler: yes). Web scraping with pandas is fun! Here we combine dataframes.
  • Chapter 6: Strings with pandas are great. It has all these vectorized string operations and they’re the best. We will turn a bunch of strings containing “Snow” into vectors of numbers in a trice.
  • Chapter 7: Cleaning up messy data is never a joy, but with pandas it’s easier.
  • Chapter 8: Parsing Unix timestamps is confusing at first but it turns out to be really easy.
The book can be downloaded here

Some Awesome Resources

<Note> All of the content, text, links, references in this post are taken from thepandas documentation website
Links and references do not represent wholesale endorsements of the resource or the author. This is not a comprehensive list,  so I welcome your suggestions and comments.
Best,
Ali Syed
I look forward to your comments below and @aaalee.

Saturday, May 2, 2015

Automate the Boring Stuff with Python

Learn to Code

cover_automate_thumbEveryone in the 21st century needs to learn to code, right? But not everyone needs to become a software engineer or computer scientist. Automate the Boring Stuff with Pythonis written for office workers, students, administrators, and anyone who uses a computer how to write small, practical programs to automate tasks on their computer.
  • Have a folder with thousands of files that need to be renamed?
  • Need to look through thousands of rows in an Excel spreadsheet looking for ones to update?
  • Have to scrape data off of several web pages?
Normally this would involve hours of tedious clicking and typing. But programming your computer to do it can save you a lot of time and effort.
Automate teaches total beginners with no programming experience the Python programming language. (Though Part 2 will be of interest to seasoned developers as well.) Python has a gentle learning curve yet is also used by professional software developers. You don't need to know all the complexities of algorithms and syntax, you just want to write basic programs to automate mundane computer tasks. In the process, even total beginners will learn to use Python to control their computers without having to learn complex information about computer science. This is a practical programming guide for the rest of us.
Automate the Boring Stuff with Python is released under a Creative Commons license. You can read it, in full, on this website. Programming is a skill that should be in the hands of everyone: No, Seriously, You Should Learn to Code..
small_coverYou can also support the author by purchasing a book or ebook: Purchase a copy ofAutomate the Boring Stuff with Python.

Table of Contents

Part 1 - The Basics of Python Programming
  1. Introduction
  2. Python Basics
  3. Flow Control
  4. Functions
  5. Lists
  6. Dictionaries and Structuring Data
  7. Manipulating Strings
Part 2 - Automating Tasks
  1. Pattern Matching with Regular Expressions
  2. Reading and Writing Files
  3. Organizing Files
  4. Debugging
  5. Web Scraping
  6. Working with Excel Spreadsheets
  7. Working with PDF and Word Documents
  8. Working with CSV Files and JSON Data
  9. Time, Scheduling Tasks, and Launching Programs
  10. Sending Email and Text Messages
  11. Manipulating Images
  12. Controlling the Keyboard and Mouse with GUI Automation
Al Sweigart is the author of Invent Your Own Computer Games with PythonMaking Games with Python & Pygame, and Hacking Secret Ciphers with Python. His books are freely available under a Creative Commons license from http://inventwithpython.com.

I400/590 : Large-Scale Social Phenomena - Data Mining Demo

I400/590 : Large-Scale Social Phenomena - Data Mining Demo

Introduction
For your mid-term hack-a-thons, you will be expected to quickly acquire, analyze and draw conclusion from some real-world datasets.  The goal of this tutorial is to provide you with some tools that will hopefully enable you to spend less time debugging and more time generating and testing interesting ideas.

Here, I chose to focus on Python.  It is beautiful language that is quickly developing an ecosystem of powerful and free scientific computing and data mining tools (e.g. the Homogenization of scientific computing, or why Python is steadily eating other languages' lunch).  For this reason, as well as my own familiarity with it, I encourage (though certainly not require) you to use it for your mid-term hack-a-thons.  From my own experience, getting comfortable with these tools will pay off in terms of making many future data analysis projects (including perhaps your final projects) easier & more enjoyable.

Hopefully you already have Python installed. If you are new to it, search around for good introductory tutorials -- I'd say it has a forgiving learning curve, comparatively speaking.

IPython
IPython is a kind of add-on for Python that brings several improvements, most importantly for us are its interactive, graphical notebooks which provide a great way to quickly develop and share code.  A gallery of interesting notebooks is provided here, including:

Libraries
In this demo, we will demonstrate the basic functionality of several useful toolboxes, including: 
  • Numpy - a library for manipulating matrices and performing linear algebra operations (matrix products, computing covariance matrices, etc.).  All the other libraries build on the data structures provided by numpy.  Here's a basic tutorial.
  • Scipy - a library that builds on top of numpy and provides more advanced linear algebra routines, statistical tools, and sparse matrices (useful in many applications, such as text mining)
  • Matplotlib - plotting library, similar in style to that used by MATLAB. Here's the official gallery (click on a plot to see the code to generate it) and here's an IPython notebook with many good examples
  • Scikit-learn (a.k.a. sklearn) - machine learning library, making it easy to do anything from PCA to training classifiers.  Here's a very quick quick start,  and here's some tutorials showing how to do common machine learning tasks.
  • Pandas - is a library for managing data sets.  It really makes a breeze of difficult or tedious tasks (loading / processing / cleaning data).  There is a 10 min intro video here, a tutorial here, another series of tutorials by Hernan Rojas.
  • Networkx - a library for working with networks.  This includes computing graph theoretic measures, laying out and plotting graphs.  A brief tutorial from the docs is here.
  • NLTK - the Natural Language Toolkit, can be very useful for text mining and language processing tasks.  Actually, we won't demonstrate it during the in-class demo, but you can see its utility in tutorials such as this andthis, as well as the documentation on the NLTK site.
You can find more useful links at Python for data analysis: the landscape of tutorials.


Data
Even simple analysis can extract interesting results from good data, but nothing can make up for bad data.  There's a few places to find potential data sets, including publicly-available data sources (Tableau software provides some data sets, and there is a directory of APIs and data sources at ProgrammableWeb).  

Another common strategy is to scrape data from the web.  There is an automatic tool to do so built by Kimono (which I haven't used but looks impressive).  Python has several tools to do this (many discussed here).  We will focus on the combination of mechanize (which simulates a browser to download HTML) and BeautifulSoup (which parses the downloaded HTML).  There are some good tutorials showing how to use mechanize and Beautiful here and here.

Installation
There are two ways to install the aforementioned tools.  The first is to use a Python distribution that already comes with all of these included, such as Anaconda.

The second option is to use the Python installer.  Once you have Python installed, run the following on the command line:
pip install -U ipython numpy scipy matplotlib scikit-learn pandas networkx nltk 
pip install -U mechanize beautifulsoup4
You may want to run this as administrator to install them system-wide.  If this options doesn't work for you, I recommend trying Anaconda.

Demo