BRIEFING: ThoughtWorks' QTB on Big Data

Some notes from this quarter's technology briefing from Thoughtworks. This session's topic was "Big Data". I was pleased that the topic was chosen as I am interest in Big Data and travel, especially how it can be used by my clients and to enhance the product that I work on.

 Caitlin McDonald from twitter has also created a Storify story from tweets during the event (with the added bonus is that I am in the background to someone's photo)

Session

The main speaker was David Elliman with Ashok Subramanian - David has also written a blog post called The Big in Big Data Misses the Point that presents some of the content covered or see the full presentation in English or German. The session started y looking at the origin of "information explosion" and how in the 1940s people were starting to get worried about the miles of shelf space would be needed by 2000 to store all the books produced. This was contrasted with the explosion of multimedia information produced now, for example pictures from camera phones or the daily output from the large hadron collider.

 Next basic architecture such as MapReduce was explained along with what makes "Big Data" - variety, volume and velocity - but also that it's what you have in your data, you don't always need a large data set. Sampling is important!

 Another point was that the relationship is that velocity and volume is inversely proportional, low volume allows more real time analysis towards batch processing with very large volumes. Sorting data takes around 80% of the time - understanding data and sorting models is key, don't expect to get a data dump and have instant insight as there is no one answer or model, it requires data scientist to understand question and go through models.

 Process described to get successful insight was:

  1. Start small
  2. Start with "?"
  3. Iteratively follow the value
Next some implementations of the Lamda architecture were described and tools discussed, finishing off with a talk about data lakes and their similarity to enterprise data warehousing - i.e. not very like big data. The quote of the day came in this session:

"Master data management is the enemy of innovation"
Overall a good session highlighting the importance of good analytics in getting insight from data. Having read their big data blog I was surprised by this as they prefer the term Big Data Analytics.

Venue

Shoreditch village hall is quite a nice venue, a little bit of a pain from Brighton and the room layout made mingling a bit harder. The sound was had a bit of feedback at the start but soon settled down. The food was great, a very tasty cheese burger (non-cheese and non-meat burgers were also available).
Location: Hoxton Hall, 193 Hoxton Street, Shoreditch, London N1 6RA, UK

Comments

Post a Comment

Popular posts from this blog

CONFERENCE: TTI Summer Forum 2017 – Getting to Grips with GDPR

Image
T his week I was at the Travel Technology Initiative's Summer Forum . The subject was the new EU (and UK!) data protection laws. These are due to come into force in May 2018. It's a large topic where the individual member state's laws and guidance are evolving. There were three main points that I picked up from the presentations. First from Dai Davis , GDPR expert at Percy Crow Davis & Co. His lively presentation talked about how a large part of the change is in how rights communicated. Before this was by "fairness" through registration of usage with a central body. The shift is to transparency by informing individuals directly. This means that the consumer mindset could then shift to match how legislation is framed. For example, with the repetition of many companies holding personally identifiable information now having to inform what they are collecting and how they are processing it. Currently, awareness isn't high and the compensation not high eno...
Read more

On HBX and online education

Image
I have written this blog post as a reflection on what has occupied me a lot in the past 3 months. After finishing my Post graduate Certificate in Technol ogy Manage ment with the O pen U niversity I was looking at what to do next in continuing my education - being a life long learner! I was looking at different online plat forms for interesting courses.  I was looking at sites like Coursera and  Future Learn , when I came across a new extension to Harvard Business School and a course they run called HBX CORe. This one paragraph pitch they provide sums it up pretty well: HBX CORe (Credential of Readiness) is a 120-150 hour certificate program on the fundamentals of business from Harvard Business School. CORe is comprised of three courses - Business Analytics, Economics for Managers, and Financial Accounting – developed by leading Harvard Business School faculty and delivered in an active learning environment based on the HBS signature case-based learning mod...
Read more

Getting email up and running for side projects

Image
This is  part of a series  about my side project  Bashfully , which aims to give graduates and other new entrants to careers a seasoned professional level way of expressing themselves through the super power of story telling. Following the core principles of being discoverable, personalised and guiding in approach. Photo by Songeunyoung on Unsplash Rectifying my previous mistake - not building community by using email engagement - has proved to be a learning experience! Since our investment is minimal (and we have no real users yet!) the tools that we use all come with some constraint/trade-off. This next bit explain this is going to be slightly technical ... With Heroku it has surfaced as putting in CNAME records in the root of the domain. With register.ly this then stops us adding any other root records, for example mail servers. Cloudfare has a service that will flatten CNAME records for you into A records so everything plays fits together. So a switch ...
Read more