These are the news items I've curated in my monitoring of the API space that have some relevance to the API definition conversation and I wanted to include in my research. I'm using all of these links to better understand how the space is testing their APIs, going beyond just monitoring and understand the details of each request and response.24 May 2017
I’m needing data for use in human service API implementations. I need sample organizations, locations, and services to round off implementations, making it easier to understand what is possible with an API, when you are playing with one of my demos.
There are a number of features that require there to be data in these systems, and is always more convincing when it has intuitive, recognizable entries, not just test names, or possibly latin filler text. I need a variety of samples, in many different categories, with a complete phone, address, and other specific data points. I also need this across many different APIs, and ideally, on demand when I set up a new demo instance of the human services API.
It is kind of a poor man’s way of creating test data, then publishing to Github for use in a variety of continuous integration workflows. I can maintain a rich folder of test data sets for a variety of use cases in spreadsheets, and even invite other folks to help me create and manage the data stored in spreadsheets. Then I can publish to a variety of Github repositories as YAML, and integrated into any workflow, loading test data sets into new APIs, and existing APIs as part of testing, monitoring, or even just to make an API seem convincing.
To support my work I have a spreadsheet published, and two scripts, one for pulling organizations, and the other for pulling locations–both which publish YAML to the _data folder in the repository. I’ll keep playing with ways of publishing test data year, for use across my projects. With each addition, I will try and add a story to this research, to help others understand how it all works. I am hoping that I will eventually develop a pretty robust set of tools for working with test data in APIs, as part of a test data continuous publishing and integration cycle.
I was profiling the New Relic API and came across their Synthetics service,which is a testing and monitoring solution that lets you "send calls to your APIs to make sure each output and system response are successfully returned from multiple locations around the world"--pretty straight forward monitoring stuff. The name is what caught my attention, and got me thinking the data and content that we run through our APIs.
Virtualization feels like it defines the levers and gears our API-driven systems, and synthetics feels like it speaks to the data and content that flows through flows through these systems. It feels like everything in the API stack should be able to be virtualized, and sandboxes, including the data and content, which is the lifeblood--allowing us to test and monitor everything.
It also seems like another reason we'd want to share our data schemas, as well as employ common ones like schema.org, so that others can create synthetic data and content sets for variety of scenarios--then API providers could put these sets to work in testing and monitoring their operations. A sort of synthetic data and content marketplace for the growing world of API testing and monitoring.
I see that New Relic has the name Synthetics trademarked, so I'll have to play around with variations to describe the data and the content portion of my API virtualization research. I'll use virtualization to describe gears of the engine, and something along the lines of synthetic data and content to describe everything that we run through it. I am just looking for ways to better describe the different approaches I am seeing, and tell more stories about API virtualization, and sandboxing in ways that resonate with folks.
I was profiling the Facebook API as part of my API Stack work. While I only use a handful of the endpoints available to me via the Facebook API, as the API Evangelist, I feel like I should have an awareness of the popular social API. Additionally, the number of great stories I find dramatically increase with the number of API profiles that I complete.
One story I extracted from my Facebook API research is about providing a dedicated test user API. Using the test user API you can add, manage and delete test users, which you can use throughout the developing and testing of your API integration. Facebook is user- centric, but it seems like the concept applies equally to any other valuable resource made available via APIs today.
I'd file this under virtualization, when it comes to organizing as part of my overall research. Providing virtualization options for API consumers is something that is only going to grow with the Internet of Things, and privacy concerns. API providers should be looking at how they virtualize entire APIs using modern approaches to containerization, so they can be used in dev, qa, and production environments, but they should also be looking at providing data and content virtualization solutions like Facebook does with a test user API.
This conversation has come up over 10 times this month, at Defrag, APIStrat, and online conversations via Skype and GHangouts. The concept of API virtualization solutions. I am not talking about virtualization in the cloud computing and Docker sense, although that will play a fundamental role in the solutions I am trying to articulate. What I am focusing on is about providing sandbox, QA environments, and other specialized virtualization solutions for API providers, that make API consumers worlds much easier.
I've touched on examples of this in the wild, with my earlier post on API sandbox and simulator from Carvoyant, which is an example of the need for virtualization solutions that are tailored for the connected automobile solution. Think of this, but for every aspect of the fast growing Internet of Things space. The IoT demand is about the future opportunity, I've talked about the current need, when I published, I Wish All APIs Had Sandbox Environment By Default.
I see envision 1/3 of these solutions being about deploying Docker containers on demand, 1/3 being about virtualizing the API using common API definitions, and the final 1/3 being about the data provided with the solutions. This is where I think the right team(s) could develop some pretty unique skills when it comes to delivering specialized simulations tailored for testing home, automobile, agriculture, industrial, transportation, and other unique API solutions.
We are going to need "virtualized" versions of our APIs, whether or not it is for web, mobile, devices, or just for managing APIs throughout their life cycle. You can see a handful of the current API virtualizations out there on my research in this area, but I predict the demand for more robust, and specialized API virtualization solutions is going to dramatically increase as the space continues its rapid expansion. I just wanted to put it out there, and encourage all y'all to think more about this area, and push forward the concept of API virtualization as you are building all the bitch'n tooling you are working on.
To Incentivize API Performance, Load, And Security Testing, Providers Should Reduce Bandwidth And Compute Costs Asscociated03 Nov 2015
I love that AWS is baking monitoring testing by default in the new Amazon API Gateway. I am also seeing new service from AWS, and Google providing security and testing services for your APIs, and other infrastructure. It just makes sense for cloud platforms to incentivize security of their platforms, but also ensure wider success through the performance and load testing of APIs as well.
As I'm reading through recent releases, and posts, I'm thinking about the growth in monitoring, testing, and performance services targeting APIs, and the convergence with a growth in the number of approaches to API virtualization, and what containers are doing to the API space. I feel like Amazon baking in monitoring and testing into API deployment and management because it is in their best interest, but is also something I think providers could go even further when it comes to investment in this area.
What if you could establish a stage of your operations, such as QA, or maybe production testing, and the compute and bandwidth costs associated with operations in these stages were significantly discounted? Kind of like the difference in storage levels between Amazon S3 and Glacier, but designed specifically to encourage monitoring, testing, and performance on API deployments.
Maybe AWS is already doing this and I've missed it. Regardless it seems like an interesting way that any API service provider could encourage customers to deliver better quality APIs, as well as help give a boost to the overall API testing, monitoring, and performance layer of the sector. #JustAThought
I was gathering my thoughts today around API management solutions can better work together, in response to an industry discussion going on between several creators of open source tooling. The Github thread is focused on proxies, and how they can work more closely together to facilitate interoperability in this potentially federated layer of the API economy, but as I do, I wanted to step back and look at bigger picture before I responded.
As I was gathering my thoughts, I also had an interesting conversation with the creator of API Garage, in which one of the topics was around how we encourage API service providers to want to work closer, but this time from the perspective of a API client workspace. This made me think, that the thoughts I was gathering about how open source proxies can better work together, should be universally applied to every step along the API lifecycle.
It should be a no-brainer for API service providers--have an API! One of the best examples of this, is with 3Scale API management--they have a very robust API, that represents every aspect of the API management layer. Other service providers like API Science, and APIMATIC, who server other stops along the API life-cycle, also have their own APIs. If you want your API tooling to work with other API tooling, have an API--think about Docker, and their API interface, and make your tools behave this way.
Provide openly licensed tooling around any service you provide. Make your tooling as modular as you can, and apply open source licenses wherever it makes sense. Open licenses, facilitate people working together, and breaks down silos. This is obvious to the folks on the API proxy thread I'm referencing, but will not be as obvious to other service providers looking to break into the market.
Speak common API definition formats like Swagger, API Blueprint, and in Postman Collections, and provide indexes using APIs.json. If consumers can import and export in their preferred definition format they will be able to get up and running much quicker, share patterns between the various proprietary services, and the open tooling they are putting to work. There are a lot of opportunities to partner around common API definitions for API deployment, and management, which would open up a potentially new world of API services that aggregate, integrate, and sync between common API platforms throughout the life-cycle.
Developers never want their tooling to be a silo. Allow the extensibility of any tooling or services, from design, to management, or client, using common approaches to connectors, plugins, etc. Make plugins be API first, so that other API service providers can easily take their own APIs, craft a plugin, and quickly bring their value to another platform's ecosystem. Plugins are the doorway to the interoperability that open APIs, source code, and definitions bring to the table for platform providers.
Open APIs, source code, definitions, and plugins all set the stage for ever deeper levels of partnership. APIs are the pipes, with open source code and definitions providing the grease, and if all API design, deployment, management, monitoring, testing, performance, security, virtualization, and discovery providers allow for plugins, strategic partners can occur organically.
All of this really sounds familiar? It really sounds like what the API space is telling the rest of the business world, so I can't help but see the irony in giving API service and tooling providers this same advice. If you want more partnerships to happen, expose all your resources as APIs, providing open tooling and definitions, allow other companies to plug their features into your solutions, and a whole new world of business development will emerge.
With the mainstream growth we are seeing in the API space in 2015, there are some pretty significant opportunities for partnership between API service providers right now--that is, if we follow the same API-first approach, currently being recommended to API providers.
I was made aware of the Citi Mobile Challenge during a conversation last week with the Anypresence team about their new JustAPIs offering. The conversation with them has triggered several potential stories, but one that stood out for me, was the API virtualization opportunities that emerge when you use solutions like the JustAPIs gateway.
To support the Citi Mobile Challenge, the bank launched a set of virtual APIs to support the event. I think their descriptions tells a lot of the story:
The APIs made available through Citi® Mobile Challenge power some of Citi's latest digital offerings around the globe. These APIs will give developers an opportunity to create solutions that could function with existing Citi technology. No customer data will be shared. Citi encourages developers to also use APIs outside of Citi to support and add additional features to their innovations.
The bank needed to mimic their production environment, but in a way that developers can still build meaningful mobile apps,without the immediate security and privacy concerns that are present when working with live systems. While I wish sandbox environments were default via all API platforms, which unfortunately is not a reality, however I think when it comes to fintech, the stakes are much higher--which smells like an opportunity for API definitions to me.
It can be a lot of work to establish and maintain both sandbox and production environments, but is something that if you do, can open up even more opportunity when it comes to supporting hackathons, providing QA environments, and much more. Additionally, if you embrace modern API definition formats like Swagger and API Blueprint in this process, as well as approaches to containerization like Docker, a whole new agility and flexibility can be realized as well.
Like API monitoring, performance, security, and other growing incentives for embracing modern API definition formats, I see the demand for API virtualization growing. When your APIs are well defined, a new world of API definition driven services emerges, making things like virtualization for hackathons, QA, load testing, simulation, as well as just a permanent sandbox possible
I have an API virtualization research area started, I just need to spend the time organizing the companies who are doing interesting things in the space, and consider some of the possible building blocks, before it is ready for prime time--stay tuned!
I was reviewing one of the many entries in my review queue of companies who are doing interesting things with APIs, and stumbled across the data visualization API—Lightning. Their implementation grabs my attention on several fronts, but their focus on delivering their API within your own infrastructure via a Heroku button, is one of the most relevant aspects.
This approach reflects a seismic shift occurring in how we deploy APIs, and how we deploy architecture overall. You need a data visualization API, let me deploy my API into your cloud, or on-premise infrastructure using popular approaches to virtualization—developers do not need to go to the API, the API will now come to you, and live within your existing infrastructure stack.
I’ve been talking about wholesale APIs a lot lately, showcasing the white label approach by some API providers, and exploring within the evolution of my own infrastructure, and as more savvy API providers jump in on this opportunity, you’ll see more stories emerge trying to understand the shift going on. Lightning is accomplishing this with Heroku, and their embeddable button, but companies who embrace a containerized micro-service centered approach to API deployment, will have a wide open playing field for buying and selling of the wholesale API driven technology being deployed across the emerging API economy.
Up until now we tend to think of APIs individually--we approach integration in terms of the Twilio API, Twitter API or the Facebook API. But as the number of public APIs has grown beyond 8K, and an unknown amount of internal and partner APIs become available, we are seeing new patterns of aggregation and interoperability emerge from companies like Singly, but also seeing automation be added into the mix by companies like Temboo, and entire backend stacks from providers like Parse.
These new aggregated or backend stacks of API driven resources can be as general as object and key-value stores, user management and other developer commodities we see backend as a service providers (BaaS) bring to the table, or they can be very personal like the photos Singly is aggregating across Flickr, Facebook and Instagram and with friends and followers across Facebook, Twitter and LinkedIn.
As I see these new aggregate and BaaS providers emerge, at an ever increasing pace, I can’t help but think--this still isn’t fast enough (its my nature, you should try being me). If I want common mobile developer resources I can adopt one of these new BaaS platforms like Kinvey or Parse, or if I need personal and social resources, which have become part of the fabric of web and mobile apps, I can go to Singly. But what if I want a more specialized network, say just for education? I might need user management, object storage, key-value storage, access to commons social tools like friends and photos across multiple social networks, but I also need access to open courseware, teacher and student directories all via a very secure, auditable, efficient stack tailored just for K-12? I will have to wait, for the next wave of startups to emerge.
While in NYC on Friday I had a great discussion with Temboo about virtualization. Not just the virtualization we’ve come to depend on with cloud computing--which is the virtualization of compute, storage, and database resources, but the virtualization of network resources and software defined networking. Companies like Nicira, Pluribus Networks, Anuta Networks, Arista, and Vyatta are emerging with new products that are allowing the virtualization of networking resource into new and meaningful network stacks for any possible implementation that you can imagine.
After these conversation with the Temboo team, over the weekend I continued to think about the potential of virtualized API stacks. Why can’t I assemble my own API stack? Why do I have to go to each API individually, or wait for new Singlys' to emerge in other verticals? Why can’t I assemble Parse, Singly, Twilio, Schoology, SendGrid into a virtualized API stack that provides not just ease of use, but the security I need to deploy a backend tailored just for K-12 education?
In this vision of the future, API providers could focus on what they do best, and not worry about every use case out there. Providers like Singly, Temboo, Parse can build abstracted layers on top of this. With this abstraction I wouldn’t be limited to just the friends and followers features on Twitter or Facebook, I could take advantage of the next generation of friend discovery tools like what Singly is delivering--in addition to the value of individual API providers.
With a virtualized approach, I could build the stack that is most meaningful for my internal, partner or public developers and if one piece of my stack is proving unreliable, I can replace it with another. APIs resources would be further commoditized, required to provide JSON definitions of their interfaces (or die a quick death), which virtual API stack platforms could use to discover and offer API resources. API ranking algorithms would emerge allowing anyone to make sure they were discovering, selecting and using the best of breed API resources in the areas that matter for any vertical.
With a virtualized API stack I could launch any specialized set of resources that I desired from the best of breed providers out there. I could blend private and public resources together and in return offer them for use in private or public environments, further blurring the lines of what is an API and how they are consumed.
As we see APIs continue to become a driving force in government, healthcare and education and bridge the online and physical worlds via our automobiles, homes, buildings and further grow within healthcare via the quantified self, potentially disrupt manufacturing with 3D printing and drive everything around us with the internet of things--the manual assembling of individual or even aggregated API stacks won’t be enough. We will need the ability to virtualize APIs stacks for any purpose within hours, not months or years.
If you think there is a link I should have listed here feel free to tweet it at me, or submit as a Github issue. Even though I do this full time, I'm still a one person show, and I miss quite a bit, and depend on my network to help me know what is going on.