Resources for understanding Transformer Architectures

The current generative AI boom is built on the foundations of the Transformer architecture used to create the large language models (LLM). The technical details of the Transformer architecture was described in the Google paper that first introduced it: “Attention is all you need“. Unless you are a trained data scientist or a machine learning engineer, the details in that paper will not make much sense to you.

There are plenty of good resources on the internet that go into details of this architecture to explain it to someone who has basic engineering background. I’d like to highlight 2 sources that I have found very helpful.

The first resource that I’d like to point you to is Jay Alammar’s blog. Jay has since expanded the blog to include lots of videos as well to explain these concepts.

The second resource that I’d like to point you to is the amazing 3 Blue 1 Brown math videos channel. Grant Sanderson has created some really intuitive visualizations to explain math concepts. He also has an entire series of videos dedicated to explaining deep learning and transformer architectures. Check out his neural network series of videos to get a good understanding of the topic.

Posted in Uncategorized | Tagged , , | Leave a comment

Security issues in SCADA networks

[This is an old published paper. I am hosting it online to benefit CiteSeer.]

Abstract:

The increasing interconnectivity of SCADA (Supervisory Control and Data Acquisition) networks has exposed them to a wide range of network security problems. This paper provides an overview of all the crucial research issues that are involved in strengthening the cyber security of SCADA networks. The paper describes the general architecture of SCADA networks and the properties of some of the commonly used SCADA communication protocols. The general security threats and vulnerabilities in these networks are discussed followed by a survey of the research challenges facing SCADA networks. The paper discusses the ongoing work in several SCADA security areas such as improving access control, firewalls and intrusion detection systems, SCADA protocol analyses, cryptography and key management, device and operating system security. Many trade and research organizations are involved in trying to standardize SCADA security technologies. The paper concludes with an overview of these standardization efforts.

Citation:

Vinay M. Igure, S. Laughter, Ronald D. Williams, “Security Issues in SCADA Networks,” Journal of Computers & Security, vol. 25, issue 7, pp. 498‐506, Oct 2006, Elsevier Publications.

Full Paper: Scada_Security_JCS

Posted in Published Papers, SCADA | Leave a comment

Analog Turbo Decoder Implemented in SiGe BiCMOS Technology

[This is an old published paper. I am hosting it online to benefit CiteSeer.]

Abstract:

Error correcting decoder’s performance is crucial for communication systems. Since the performance of Turbo codes is close to the Shannon limit, ultra high-speed turbo decoder is desired. The aim of this design is to demonstrate the feasibility of implementing high-speed Turbo decoders in analog VLSI using SiGe BiCMOS technology. Detailed conceptual design is presented. Simulation results show higher performance than any known decoders in term of decoding speed. Testing plans are also presented.

Citation:

W. Huang, Vinay Igure, G. Rose, Y. Zhang, M. Stan, “Analog Turbo Decoder Implemented in SiGe BiCMOS Technolgy”, Honorable Mention Winner (Fourth Position) at the 40th Design Automation Conference (DAC) Student Design Contest, June 2003; Also selected for Poster Presentation at the ISSCC 2004; Selected among the top 15 teams in the first Phase of the Semiconductor Research Corporation’s (SRC) SiGe Design Contest 2002 ‐2003

Full Paper: UVA_Turbo_DAC

Posted in Published Papers | Leave a comment

A Defence against “Sleep Deprivation” Attack

[This is an old published paper. I am hosting it online to benefit CiteSeer.]

ABSTRACT
Many current and future embedded computing devices will operate from severely limited  battery power, and they will conserve this power by spending most of their time in a very low power “sleep” state. These devices can be subjected to a denial of service (DoS) attack that prevents them from entering the sleep state. This “sleep deprivation” attack is accomplished by an attacker repeatedly sending false messages until the limited battery power of the attacked device is exhausted. A defense against this attack uses a very low power authentication scheme that must be passed before the main processing elements of the sleeping device will be awakened. An implementation of this low power authentication scheme is discussed along with consideration of the broader authentication issues for the embedded computing devices.

Citation:

Vinay M. Igure, T. N. Blalock, R. D. Williams, “A Defense against ‘Sleep‐Deprivation’ Attack,” IASTED International Conference on Communications and Computer Networks (CCN2002,) November 4‐6, 2002 MIT Cambridge, Massachusetts.

Full PDF of Paper: Sleep_Deprivation

Posted in Published Papers | Leave a comment

Big Data and Graphical Representation

Big Data is expected to play an important role in the Smart Grid. As we have more devices monitoring and collecting information regarding our energy consumption behavior, the amount of data will increase. This data can provide lots of insights. When we present this information to consumers it matters not only what we present but also how we show that information.

One of the classic books regarding graphical data representations is Edward Tufte’s ‘The Visual Display of Quantitative Information. After a long time, I finally managed to knock this book off my reading-list. The best insight in the book is one of Tufte’s Principles of Graphical Excellence:

“Give the viewer the greatest number of ideas in the shortest amount of time using the least ink and space”

This one sentence captures the essence of the book. We can even derive his other principles using the above principle as a sort of ‘fundamental rule’. In order to give the viewer ideas, the data has to be meaningful (have substance) and must be in support of a theory. Well-designed graphics attract our attention and compel us to contemplate on the data.

A lot of graphics that we see in popular media is dumbed down. The writers assume that the readers are not capable of understanding statistics and math and attempt to over-simplify things. This leads to erroneous graphics that mislead people. Tufte has devoted an entire chapter on ‘Graphical Integrity’. Once you read this book, you will never look at graphics in the general media without double-checking them for accuracy. Tufte presents plenty of examples of how even well-intentioned graphics can mislead us and present us with wrong information.

How do you decide if a graphic is good? Is it possible to measure graphics for excellence? Tufte describes the concept of ‘data-ink’ to distinguish between good and bad graphics. The concept is very simple and is again related to what I called his fundamental rule. In order to use the least ink, the graphic must try to utilize a greater percentage of the total ink used on the graphic to show data. In other words, the graphic must eliminate all non-data related ink on the graphic. In today’s digital publishing world, ‘ink’ might sound outdated, but the concept is certainly relevant for producing great graphics. Tufte walks through many examples and shows how graphics improve by maximizing the data-ink ratio.

On a more contemporary note, if you happen to read Tufte’s book, I also highly recommend that you check out Bret Victor’s ‘Magic Ink’. While Tufte’s work is concerned with the general graphical representation of data, Victor’s work is concerned specifically about developing good graphical user interfaces in software programs. The main takeaway from Victor’s paper is his classification of software programs. People use software programs for essentially three tasks: to Learn, to Create and to Communicate. Understanding the main reason why customers use your software can guide you to build better graphical interfaces.

In today’s world, Big data, or any data for that matter, will ultimately be condensed into user-understandable format in software. Therefore it is essential that product developers have a good sense of the issues involved in presenting data in software programs. Tufte and Victor present a great starting point to understand these issues.

Posted in Uncategorized | Tagged , , , | Leave a comment

A Recap

It’s been almost 9 months since I started writing this blog. Although I have written only 25 articles so far (some of them not long enough to call ‘articles’), I think the blog has gone off in too many directions. I initially wanted to focus only on technical issues. But I seem to have written a lot of non-technical stuff. Here’s the complete list so far:

By the way, I borrowed this idea of a blog recap from Ribbon Farm. Blogs tend to get lost pretty soon and the archive lists are not very user friendly. These recaps help bring some stuff back into the open.

I can broadly classify the articles so far into three categories:

  1. Smart Grid
  2. Product Management/Management/Marketing
  3. Technology Trends and their effects on our lives

I was quite surprised by the feedback I have received to some of the articles. The ones in bold above are the ones that generated the most feedback. Going forward I plan to concentrate on technical topics. The strange thing is that I have not yet touched upon some of the topics that I initially wanted to write about. Our lives are so integrated with technology that it is hard to ignore the affects and think about it. If I come across some issues on which I can add a new perspective, I will still write about them. But for the next few months, I will concentrate on the technical side of Smart Grid communications.

Posted in Uncategorized | Leave a comment

Creating Great Products

There are quite a few lists of good Product Management books on the Internet. Marty Cagan’s “Inspired: How to Create Products Customers Love” often shows up in such lists and that’s how I came across it.

The most useful part of this book is Cagan’s list of ‘Opportunity Assessment’ questions. The complete article is even available on his website. In fact, the book definitely feels like a collection of blog articles. However, that definitely does not diminish the good insights in the book.

The second take away from this book was the concept of organizing your product design process around a core set of ‘Product Principles’. As mentioned in the book, having this set of principles can be very useful for resolving conflicts between people who disagree over some of the features in the product.

The third useful point was the concept of designing products around emotions. This is especially important if you are in the B2C space and are building products for consumers. It is so much easier to sell a product to a customer if he is emotionally connected to the product. On a related topic, check out Simon Sinek’s great talk about why ‘People Don’t Buy What you do, buy why you do it’.

Well, that’s all I have to say about the book. When I started this blog, I was planning to write regularly about the books I read. I soon realized that that was a difficult task. The rate at which I read books will always be greater than the rate at which I can write about them. And more importantly, not all books are worth writing about. Unless I read something that truly blows my mind and compels me to write about it, this will be the last book related post for some time. I just wanted to put this out ther and move on to other topics.

 

Posted in Product Management | Leave a comment

Information & Meaning

“The central dogma says, “Meaning is irrelevant.”, Freeman Dyson on Shannon’s Information Theory

“Information is cheap, but meaning is expensive.”, George Dyson

Of course, the context of both these quotes is completely different. I happened to come across both of them quite recently and found it quite interesting to compare the quotes from father and son.

[I kind of cheated on this post. I haven’t written in a while and I wanted to post something quick and easy so that I can start getting back in the groove.] 

Posted in Uncategorized | Tagged | Leave a comment

The Producing-Consumer Paradox

In a previous post, I had briefly touched upon the idea of living in a world where the amount of content produced, both print and media, far exceeds what one can consume in a lifetime. I had come across NPR’s blog on that issue and the concept fit well with my thoughts about a book I was reading at that time.

In today’s digital world, everyone seems to be producing content. There are millions of blogs, photo collections, and online videos; thousands of TV channels; thousands of ebooks available at the touch of a button; millions of other books in the old-fashioned paper format; etc. And adding to this is the constant stream of updates from the social media networks: a few million tweets are generated every second and a few billion Facebook status updates every day. Judged purely by numbers, we are living in humanity’s golden age of creation. Never before in history have so many people contributed so much content for global consumption.

Strangely, this explosion of content has happened in parallel with the rise of a new class of computing devices: the purely consuming devices such as tablets and smart phones. These devices are great for reading, listening to music, watching videos and online shopping. In short they are great for consuming content but not for producing anything of significance. The most productive things we can expect from these devices are short emails, 140 character tweets or Facebook status updates or photo/video uploads.

Microsoft has been openly experimenting with tablet computing for a very long time. It has had a string of failures in this space. Apple came out with one product, the iPad, and conquered this market. One of the reasons, and I think it’s a big one, for this success was because Apple decided to ignore the content producers and made a product aimed primarily at content consumers. Microsoft was having so much difficulty with tablets because it was struggling to figure out how a tablet could be used in the office environment by someone who needed to work on it for 8 hours a day. It was and it still is difficult to design such a machine. Apple’s brilliance was that it totally ignored this market and concentrated only on the couch-surfer.

By concentrating on the couch-surfing consumer, Apple created a product ideal for lying back and consuming content. Take a look at the iPad unveiling event: Notice Steve Jobs uses the tablet by sitting on the couch and leaning back. Back when Windows Vista was released, Bill Gates made a promotional interview on the Daily Show. On that show, Jon Stewart asked Bill Gates how the PC, a device that requires users to lean-forward to use it, would ever replace the TV, a device that was perfect for lying back and relaxing. Bill Gates did not have a good answer at that time, but the iPad certainly made the leap from leaning-forward to lying-back computing devices.

The Paradox

So on one side we have more and more people using purely consuming devices and yet on the other side, we have an explosion of content. I term this phenomenon the ‘Producing-Consumer Paradox’. How is it possible to produce content from devices that are only great for consumption? There are two reasons for this:

  1. The digital world is split between content Producers and content Consumers. The producers use devices that are better suited for production while those that are not interested in contributing content use the consuming devices. There truly are greater numbers of people actively working to create good content. The digital revolution has made it extremely easy to publish your content to a wide audience and many people are seizing this opportunity.
  2. A significant portion of online content is trivial stuff that does not merit one’s attention. I include the vast majority of social network content in this category. A lot of such content can be easily produced by the primarily consuming devices.

The reason number 2 above, explains why I call this the ‘Producing-Consumer’ and not ‘Consuming-Producer’. The ease of production of trivial content provides a sense of contribution to people who are largely consuming content.

This situation raises two important questions. On the producing end, how do you make your content stand out? On the consuming side, how do you distinguish between good Producers and the mediocre or bad Producers?

The most important rule for content Producers: Respect your Consumer’s Time

If you are in the Producer camp, take a moment and think about this: Out of all the billions of other items that a consumer could spend time on, she has not only discovered your content but also decided to spend her precious time consuming your content. What are the odds of this happening? If you are part of well-established media, the odds for your content to be consumed are good but they are decreasing with each passing day. If you are a blogger or an independent video producer on YouTube, the odds of someone consuming your content are extremely low. So when that rare consumer does visit your site, what do you offer them? Is your website just full of keywords to fool a search engine or do you actually have content that someone might appreciate? The least that you can do is to respect their time. Make that extra effort to produce good content that your readers or viewers will appreciate. Of course, it is never possible to know what will please a person, but making that little bit of extra effort to respect their time will improve the quality of your content.

At about the same time as I write this, Facebook announced it’s new ‘frictionless sharing’ concept that aims to flood people with even more content. According to Faceook, every person’s every online activity merits your attention. “Raj just read a news article. Here it is.” “Raj just watched a video of a cat. Check it out.” The next time you feel the urge to contribute to this endless stream, take a moment and think: Is this really worthy of someone’s attention? Slate’s Farhad Manjoo has great take on this problem: if everything is shared instantly, then how do we distinguish between content that is worthy of being shared and unworthy content?

I once attended a technical talk and I don’t remember what it was about and who gave it. But I do remember the advice the speaker gave us: If you sit through a talk for about an hour and learn at least one new thing, the talk was worth your time. I didn’t learn anything technical from that talk but I did learn that valuable lesson. If you consider yourself a serious content producer then aim to follow that principle. Write your blogs, your papers or create your videos/audio such that you add some value to someone’s life by at least a minor amount. And I repeat, it is never possible to know if your content is really good but as long as you produce with the intent of respecting your consumers’ time, it will have an effect on the quality of your content.

What about the Content Consumer?

As a regular consumer, how do you deal with the flood of content? You can either succumb to the deluge, waste your time with trivial content or you can seek out the stuff that will enrich your lives. I am not in a position to advice anyone on how to deal with this problem. Being aware of the problem is probably a good beginning.

Are we living in a post-PC World?

There is a notion that we have now entered the post-PC world and that somehow PCs are no longer relevant. There can be no doubt that smart phones and tablets have definitely made it easier to do many of the things for which we previously needed PCs. However, I firmly believe that any sort of serious content generation will always require some sort of a PC. I say ‘some sort of a PC’ because by PC, I refer not to a Microsoft Windows based machine, but to any device that provides a user interface that is conducive to working for long periods of time. For now, this means that the minimum requirements are a keyboard, a mouse and a screen that can rest on its own. I could never write this blog on a smart phone. I have not yet used a tablet but I sense it would probably be difficult to write long posts without an additional keyboard. The current crop of post-PC devices seem to be fit only for trivial content generation. Of course, they are great for passive content consumption. Enjoy!

Posted in Uncategorized | Tagged , , | 3 Comments

Networked Lighting

In a previous post, I had written about the incandescent light bulb and its gradual demise. If the incandescent light bulb is disappearing, then what product will take its place to illuminate our lives? It looks like the future of lighting is going to be the LED. Wired magazine recently had an article about the ongoing innovations in the lighting industry. The LEDs appear to be the clear winner in the battle between CFLs and LEDs. Although the current price tag of LED bulbs is way too high, it is expected to come down a lot in the next few years. There are significant technical challenges to overcome before the LED bulbs become common-place.

The thing that stood out in that article was the note about manufacturing process of LED bulbs. LEDs are after all semi-conductor diodes. They are manufactured similar to all other semi-conductors products. This means that the core of our new lighting products will be manufactured in the same process that manufactures our digital networked world. This fact ties in well with another recent announcement.

A couple of months ago, NXP semi-conductors announced their Green Chip Lighting technology. Their technology is available for CFL and LED bulbs. Note that this technology is currently an add-on to the light bulb. This means that the networking technology is added on to the lighting technology. A possible improvement to this technology would be to integrate the lighting and the networking components. The LED and networking technology are manufactured in the same process, so it might be possible to integrate both of them on the same chip. This might lead to the development of something along the lines of a ‘Networked-Light-On-a-Chip’. I am not sure what it would take to make this idea a reality but such a product would be really cool. Networked lighting technology would reduce energy consumption in two ways: the LEDs consume a lot less energy to generate the same amount of light and the networking features makes it easy to manage energy consumption in large buildings.

It has been a long journey for CFLs and LEDs to replace the incandescent light bulb. Fluorescent lighting was invented ton the 1890s and LEDs have been around since 1920s. However there is one part of our modern electric lighting that will be even harder to disrupt: the standard Edison socket. Every new lighting technology that has been invented has tried to make bulbs that fit into the Edison socket. The major problem with trying to adapt both the CFLs and LEDs to our lighting needs was in trying to make these technologies fit in to the socket. If LEDs and networked lights truly change the way we light our worlds, they would need to disrupt the Edison socket. This might be a bigger challenge than integrating lighting and networking technologies.

 

Posted in Smart Grid | Tagged , , , , | Leave a comment