This post highlights an F# feature called type providers that makes my life as a software developer easier. Ever had to parse some JSON? Explore a shiny new REST API? Maybe you’ve had to quickly try out an undocumented SOAP service?
Imagine you could talk to any kind of service or interact with any kind of data in your regular IDE, using the full capabilities of a static type system with minimal fuss. No need to generate proxy client code from .wsdl files, no need to reference any Nuget package. With some help from F#’s type providers, It just works™!
Introduction
I’ve been looking to introduce F# in my daily workflow. As a consultant, I often don’t get to pick the programming language I work in. As a result, my experiences with F# are limited to side-projects or throwaway code that doesn’t get committed to version control. One area where F# and its ecosystem are really practical and even outperform alternatives is in exploring data and services, both in the broad sense of the term. Say you have to integrate with a third party REST API that provides little-to-no documentation. F# provides a magical feature called type providers that allows you to explore this API with static types, minimal fuss, and all the features you’ve come to rely on of your trusty IDE. It’s the perfect way to get started with functional programming in a .NET environment.
Demystifying F# type providers
So, what exactly are these so-called type providers? Talking to any service or data source in any data format in a strongly typed way, without even referencing so much as a third-party assembly or having to write a single line of reflection-heavy plumbing code. That all sounds magical, doesn’t it? This is one of the features type providers enable.
The official documentation defines a type provider as follows:
An F# type provider is a component that provides types, properties, and methods for use in your program. […] The types provided by F# type providers are usually based on external information sources. For example, an F# type provider for SQL will provide the types, properties, and methods you need to work directly with the tables of any SQL database you have access to. Similarly, a type provider for WSDL web services will provide the types, properties, and methods you need to work directly with any WSDL web service.
Whelp, that’s all a bit abstract and not really enlightening is it? The basic idea is this:
- You pick a source. This can be a URL to a SOAP service, a connection string to a SQL database, some sample JSON data, or whatever you want to explore.
- You provide this source to a specific type provider. The type provider will interpret the source and generate all the types you need to interact with it in a statically typed way.
- You can now use these generated types to explore the source. Batteries included!
There’s all kinds of type providers out there. Bear with me for a couple of concrete examples below. If you search for “<your source> F# type provider” chances are it exists. If not, you can always create your own type provider.
Still too abstract, I hear you say. Right, on to some examples! You can find the full code for all these example on github.
Example: converting a CSV file to JSON format
Last week I had to upload some data from a comma-separated CSV file to a REST API. Usually this involves writing code to read from a file, parse the CSV rows and transform them to a JSON representation. Enter a world of counting indexes, stringly typing everything and debugging until you get something that looks like half-decent output. Right?
There’s a couple of type providers that really helped me out with this task: CsvProvider & JsonProvider.
First, let’s parse the CSV file and read all rows:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #r @"..\packages\FSharp.Data\lib\net40\Fsharp.Data.dll" | |
| open FSharp.Data | |
| let [<Literal>] csvPath = ".\goodreads_library_export.csv" | |
| type File = CsvProvider<csvPath> | |
| let excel = new File() | |
| excel.Rows | |
| |> Seq.sortByDescending (fun book -> book.“Average Rating“) | |
| |> Seq.take 10 | |
| |> Seq.map (fun book -> book.“Average Rating“, book.Title) | |
| |> Seq.iter (printfn "%A") |
So what’s happening here?
- Lines 1-2: reference the FSharp.Data dependency that contains definitions for the CSV type provider and opens the module.
- Line 4: is a path to a CSV file. This is the source for the type provider.
- Line 5: provides the source to the CSV type provider. Now the type provider will generate all the types you need to access this CSV file in a statically typed way.
- Lines 6-11: is just me exploring this CSV file. Don’t be fooled by all the quotes in the property names, all the properties I’m accessing are type-checked and my editor provides full IntelliSense support. No more messy indices and off-by-one errors, I can just use the column names in my code. The names of the automatically generated properties are based on the column headers in the actual CSV files. The types of the columns are derived from the data in the actual CSV file. Magical!
Now we can read data from a CSV with static typing in a couple lines of code, how do we convert this data to JSON? That’s where JsonProvider comes in. You can use this type provider for a myriad of things, but one of its use cases is generating types based on an example JSON document you provide. All you need is a small sample of the end result so it can infer the schema.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #r @"..\packages\Fsharp.Data\lib\net40\Fsharp.Data.dll" | |
| open FSharp.Data | |
| [<Literal>] | |
| let sample = @"{ | |
| ""title"" : ""Management 4.2"", | |
| ""author"" : { | |
| ""firstName"" : ""Jurgen"", | |
| ""lastName"" : ""Appelo"" | |
| }, | |
| ""ISBN"" : ""978-0321123479"", | |
| ""createdAt"" : ""2017-04-07T23:03:52.692+02:00"", | |
| ""updatedAt"" : ""2017-06-09T13:44:31.728+02:00"" | |
| }" | |
| type BookTypes = JsonProvider<sample, RootName="book"> | |
| let book = BookTypes.Book("title",BookTypes.Author("first name", "last name"),"isbn", System.DateTime.Now, System.DateTime.Now) | |
| printfn "%A" book |
We’re basically doing the same thing here:
- Line 4: define some sample JSON as a source.
- Line 14: provide this source to the JSON type provider. This will again generate all the types you need to interact with the source. In this case, it will generate types that represent the JSON schema.
- Lines 15-16: I’m using the generated types to build a JSON object. Note that the type provider even generates types for nested objects (like the author) and infers the types of all properties based on the sample data.
So, let’s now tie these two type providers together so we can read from a CSV file and transform the rows to JSON objects:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #r @"..\packages\FSharp.Data\lib\net40\Fsharp.Data.dll" | |
| open FSharp.Data | |
| type BookTypes = JsonProvider<"sample-books.json", RootName="books"> | |
| type GoodReadsExport = CsvProvider<"goodreads_library_export.csv"> | |
| let toAuthor (lastNameFirstName : string) = | |
| match lastNameFirstName.Split([|", "|], System.StringSplitOptions.RemoveEmptyEntries) with | |
| | [| lastName; firstName |] -> BookTypes.Author(lastName, firstName) | |
| | _ -> failwithf "Let's not deal with errors today." | |
| let export = new GoodReadsExport() | |
| export.Rows | |
| |> Seq.sortByDescending(fun r -> r.“My Rating“) | |
| |> Seq.take 20 | |
| |> Seq.map (fun row -> | |
| BookTypes.Book( | |
| row.Title, | |
| row.“Author l-f“ |> toAuthor, | |
| row.ISBN13, | |
| createdAt = row.“Date Added“, | |
| readAt = row.“Date Read“)) |
And there you have it, strongly-typed conversion from CSV data to JSON output in only a couple lines of code.
This first example shows the power behind F# type providers. They enable easy interaction with stuff like CSV data, web services or whatever else you can think of in a strongly-typed way without having to write any boilerplate code. Now that we have the basic idea covered, let’s dive into some more interesting examples.
Example: talking to a WCF service
I’m currently working in a context that makes extensive use of WCF services. There’s a lot of tooling out there to support interaction (svcutil.exe, WCFStorm, Linqpad,…). They’re all great and have their own benefits, but there’s no one magical solution that offers all of the following:
- I want to work in (and use the full power of) whatever IDE I’m using to develop the WCF service. I don’t want to continually context switch.
- I want tor write actual code that I can commit to source control and share with my team.
- It has to just work. I don’t want to bother with publishing and referencing Nuget packages and/or binaries.
- I want really fast feedback. Running a unit test is fine, but just selecting some code and pressing <alt+enter> is even better! If you’re new to F# or functional programming in general, take a look at the concept of a REPL. F# interactive (fsi) is F#’s REPL. It’s basically Linqpad on steroids and provides a really tight feedback loop.
I recently discovered the WsdlService type provider that made me even more productive in contexts like this. The concept remains the same:
- Use a SOAP endpoint as the source. I’m using a public SOAP service to not disclose any client code. But trust me, this works perfectly fine with WCF services as well.
- Provide this source to the WsdlService type provider. This will generate all the types you need to interact with the service (find out here how this black magic works behind the scenes).
- Explore the WCF service in a safe manner, with static typing.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #r @"..\packages\FSharp.Data.TypeProviders\lib\net40\FSharp.Data.TypeProviders.dll" | |
| #r @"System.ServiceModel.dll" | |
| #r @"System.Runtime.Serialization.dll" //Needed to see the generated contract types in you IDE. | |
| open FSharp.Data.TypeProviders | |
| open System.ServiceModel | |
| type Service = WsdlService<"http://webservices.oorsprong.org/websamples.countryinfo/CountryInfoService.wso?WSDL"> | |
| let svc = Service.GetCountryInfoServiceSoap12() | |
| let response = svc.FullCountryInfo("NG") | |
| printf "Languages of %s: " response.sName | |
| response.Languages | |
| |> Seq.map(fun lang -> lang.sName) | |
| |> String.concat "," | |
| |> printfn "%s" |
- Line 7: defines the source (a URL to a wsdl) and provides it to the WsdlService type provider. Again, the type provider will automatically generate all the types you need to interact with the source.
- Line 8: gets a WCF binding to the actual WCF service. I can now use this reference to talk to it directly.
- Line 10-16: explore the source. Note that both the operation name as input- and output types are all statically typed.
For me this beats all other alternatives for interacting with WCF services out there: I get to write actual code. In my IDE of choice. I have a very tight feedback loop. I don’t need to reference any third-party dependencies. I don’t have to context-switch. Well, except maybe the programming language if I’m not using F# for production work. For me personally it’s been worth the effort to learn F# just for this purpose.
We’ve now seen how the WsdlService can make you super productive if you’re working with WCF or other SOAP services. But that’s not what the cool kids are using nowadays. What about type provider support for REST services?
Example: exploring a REST API
So now we can talk to SOAP services with an explicit contract. How about a loosey goosey REST API? The previous examples should give you a good idea where this is going. In the JSON parsing example above, we only scratched the surface of what we can do with the JsonProvider. With this type provider, we can easily explore REST API’s as well. Just point the JsonProvider to an active REST URL and it will generate types on the fly based on the sample JSON the endpoint returns.
This example is based on evelinag’s F# wrapper for the Star Wars API (swapi). swapi provides a REST API for all things Star Wars. Let’s use this API to go over all the people in the Star Wars universe, find Darth Vader and look up which movies he starred in:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #r @"..\packages\FSharp.Data\lib\net40\Fsharp.Data.dll" | |
| open FSharp.Data | |
| open System | |
| //Easily talk to a REST API. | |
| let [<Literal>] URL = "http://swapi.co/api/" | |
| let [<Literal>] PeopleUrl = URL + "people/1" | |
| let [<Literal>] PagePeopleUrl = URL + "people/" | |
| let [<Literal>] FilmUrl = URL + "films/1" | |
| type SWAPI = JsonProvider<URL> | |
| type PagePeople = JsonProvider<PagePeopleUrl> | |
| type People = JsonProvider<PeopleUrl> | |
| type Film = JsonProvider<FilmUrl> | |
| let root = SWAPI.GetSample() | |
| let buildUrl url id = sprintf "%s%s" url id | |
| let loadPeople (pageUrl) = | |
| let rec iter (pageUrl : string) = | |
| seq{ | |
| if (pageUrl <> "") then | |
| let currentPage = PagePeople.Load(pageUrl) | |
| yield! currentPage.Results |> Array.toSeq | |
| yield! iter currentPage.Next | |
| } | |
| iter pageUrl | |
| let allCharacters = loadPeople root.People | |
| let withFilms = allCharacters |> Seq.map (fun p -> p, p.Films |> Seq.map Film.Load) | |
| let (_,filmsStarringDarthVader) = withFilms |> Seq.find (fun (p,films) -> p.Name.Contains("Darth Vader")) | |
| filmsStarringDarthVader |> Seq.map (fun f -> f.Title) |> Seq.toList |
This snippet uses some more exotic F# syntax to handle lazy loading & server-side paging, but the idea is the same. Just point a type provider to a source which will generate some types and start exploring!
And before you lose any sleep over it, the result is [“The Empire Strikes Back”; “Revenge of the Sith”; “Return of the Jedi”; “A New Hope”]. You’re welcome.
Conclusion
F# is great, but sadly I can’t always use it for production code. I can however always use it as a productive tool to explore data and API’s in a type-safe and interactive way using type providers. With just a couple lines of code you can start exploring, no need for any of the boilerplate. You get immediate feedback using the fsi REPL. You get all the static typing benefits for free, even for something as dynamic as a REST API.
One caveat though: I wouldn’t use these type providers for production code. Seeing that compilation of F# code that uses a type provider often requires an active internet connection so it can connect to the source and retrieve sample data, I don’t see this play nice in an environment where services go down all the time and build servers have limited internet access. But if you’re just exploring stuff, by all means go for it.
Further reading
- Has this gotten you interested in functional programming in F#? Scott Wlachsin has a brilliant website called fsharpforfunandprofit, which contains everything you ever want to know about functional programming in F#, from absolute beginner to OHMYGOD THE MONADS.
- Definitely take a look at how to get started with F# in your daily work if you want to start using F# but are not currently writing production F# code.
- Already have some experience with F# but still not feeling very productive? Mathias Brandewinder has some awesome tips for scripting with F#.
jvaneyck 