Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Generative Programming Methods, Metaprogramming and Language Oriented Programming (LOP) books

News

Selected Computer Books Best Compiler Construction Books XML
Static content generators Beautifiers and Pretty Printers Compilers Algorithms Lexical analysis
Recursive Descent Parsing A symbol table    
Program Generators with XML and Java Generative Programming Methods, Tools, and Applications Code Generation in Action Programming Language Processors in Java: Compilers and Interpreters

Introduction

The term generative programming (also often called metaprogramming or better, but slightly longer "Language oriented programming" (LOP)) is a fuzzy term that encompasses almost any automatic source code creation.

That included old case of generating and executing code within he program itself (possible in assembler, LISP,  Unix shell and most scripting languages).  that idea of modifying machine instructions within assembler program is as old as computers themselves (so called self modifying programs)

But graduation to more "politically correct" CS subject happened much later when the idea of compiler compiler  became fancy for a short time. The first wide scale commercial application was macro generation of assembler code. For example the kernel of OS/360 using sophisticated macros of IBM macro processor (which in itself is a computer classic).  This key idea is to make a tailor-made program for particular hardware configuration.

Then it found another important practical applications -- bootstrapping of compilers -- moving the compiler from machine architecture to another one. For example cross compiler generated code for a different architecture which then is used to compiler the compiler on this architecture.  Typically the resulting code is not machine instructions but the assembler as the translation needs to be only once and efficiency is not an issue. Also it is easier to find bugs if you have assembler version of the code.

There are probably dozens terms for the same concept. For short lived "compiler-compiler" fashion period the next time the term resurfaced in slightly different context as a part of (now also forgotten) idea of  Computer Aided Software Engineering (CASE). which generated quite a movement (actually it brought into forefront syntax coloring in editors, and gave a strong impulse in creation of first IDE for programming languages). But eventually this boom also faded.

Now this term is use mostly in a rather narrow context of XML transformation, generation of code for .Net and generation of SQL. The often used terms are  "source code generation"  or Source-to-source compiler.

Benefits of the method are not automatic. They are limited to a broad but still limited class of application and first of all anything related to text transformation. Like with everything it rewires certain ingenuity and practice as well as the ability to see benefits and drawbacks of this approach for particular task (compare this with OO "for hammer everything is a nail" approach ;-). If used wisely it can produces innovative and very high quality programs.

Simple example

Here simple example of a generative program is Unix shell (in this particular case this is just generation of a set of alas commands from the files that contains frequently used locations. the generated file later can be sourced from $HOME/.quick_aliases

#!/bin/sh
# Nikolai Bezroukov, Jul 1, 2010
# quick_aliases.sh generate set of aliases for files in $HOME/favorites 
#
while read file ; do
   fn=`basename $file`
   echo "fn=$fn"
   if [ -f $file ] ; then
      echo "alias 4$fn='view $file'" >> $HOME/.quick_aliases
   elif [ -d $file ] ; then
      echo "alias 2$fn='cd $file'" >> $HOME/.quick_aliases
   else
      echo $file is nether regular file nor directory
   fi
done < $HOME/favorites
then you can invoke it  the following way:
 $HOME/bin/quick_aliases.sh # generates set of quick access aliases
 . $HOME/.quick_aliases # inject generated aliases into the current environment

"Object code" that such a generator produces can be written in any suitable scripting or compiled language. Often generating program for specific case can dramatically simplify the code and speed up execution.  From this point of view it can viewed as a compile time optimization. It also can serve as a program simplification tool. for example instead of coding all parameter processing for a particular script you can write a generator that simply modifies source given a set of option. This is way the main program became much simpler and more manageable (often processing of option is longer then script itself ;-)

Recommended books

One of the few books that treat this method as a new programming paradigm was Code Generation in Action by Jack Herrington published in 2003:

Code Generation in Action by Jack Herrington

[Jul 1, 2003] Code Generation in Action by Jack Herrington

This is one one the few books dedicated to this subject. and probably the most readable.
Note: His site codegeneration.net disappeared, so it looks like the interest of the author toward code generation was short lived. Right now this domain is used for code generation conference. The book is pretty superficial overview of some generational techniques. It's broad but not deep. But if we exclude strictly compiler oriented literature this is one of the few that at leqst somehow illuminate the subject. 
From Amazon review: "All the code snippets and regular expression samples are explained in a very clean, detailed way. I was pleased to see that many examples were non-trivial, covering concrete, real world, implementations. Herrington uses Ruby as an implementation language; you may like it or not, but what really matters here are the concepts, not the syntax and even if you don't know Ruby (like me), you will find the code pretty easy to follow. "
Code Generation in Action is an A-to-Z guide covering building, buying, deploying and using code generators. If you are a software engineer-whether beginner or advanced-eager to become the "ideas person," the mover-and-shaker on your development team, you should learn CG techniques. This book will help you master them.

What's Inside:

Includes generators for: Over his twenty years of development experience, Jack Herrington has shipped many software applications helped by code generation techniques. He runs the Code Generation Network

Midwest Book Review on November 13, 2003

5.0 out of 5 stars Avoiding common mistakes to generate high-quality code

Expertly written by Jack Herrington (a software developer of twenty years' experience), Code Generation In Action is a competent and "user friendly" guidebook and self-teaching tool to the art and science of building and using programs to write (or assist in the writing of) other programs.

Avoiding common mistakes to generate high-quality code free of errors, and especially to create quick and easy code for the most repetitive or "gruntwork" tasks of a programming project are the core features of this invaluable and highly commended resource and reference which informatively covers code generation basics, CG techniques, patterns of CG design, how to deploy generators, numerous examples of generators, and so much more.

Kannan Narayanan from CA United States

5 out of 5 stars Good book with the right perspective and plenty of examples, October 30, 2003

The writing is precise and clear with annotated examples everywhere. Positioning and justification of various techniques is very compelling.

The author's classification of various forms of active code generation clearly elucidates the potential of Code Generation. Usage of templates for code generation is an excellent suggestion. Explanations on various code snippets and regex macros are simply second to none. Chapters 3, 4, 5 and 10 are a must read for every developer.

Having implemented a large-scale database conversion from IDMS to DB2 (schema, data dictionary, run time and programs) using home grown automated generators in the past, I really enjoyed reading Chapter 10. I completely agree with the assertions made there and I am impressed by the way the author addresses common concerns. This chapter documents a practical approach to ease the burden of writing repetitive code for code heavy frameworks. Schema Oriented Code Generation is a practical approach to code generation. I also find various references in this Chapter and others extremely valuable.

The author has shown that with sufficient metadata about a system, a significant portion of the repetitive coding tasks relating to data access, user interface, test and documentation can be automated in a consistent manner using custom code generators.

It is refreshing to see code snippets in Ruby. The author's selection of Ruby becomes self evident after reading the various code snippets.

I find this to be a very compelling book and a must have for architects and seasoned developers!

Foti Massimo (see more about me) from Savosa Switzerland

5 out of 5 stars Finally a dedicated book on the subject, November 29, 2003

I embraced code generators long time ago, so I was more than happy to finally found a dedicated book on the subject, since this is almost unexplored topic. The author clearly has a solid understanding of the topic, and manages to deliver a compelling book, with an excellent flow, where each chapter builds on previous concepts and ideas.

All the code snippets and regular expression samples are explained in a very clean, detailed way. I was pleased to see that many examples were non-trivial, covering concrete, real world, implementations. Herrington uses Ruby as an implementation language; you may like it or not, but what really matters here are the concepts, not the syntax and even if you don't know Ruby (like me), you will find the code pretty easy to follow.

John Wallaceon October 29, 2003

I purchased Code Generation in Action a few weeks ago and just couldn't put it down!

I have been a developer for 20 years now and encountered code generation for the first time back when I was writing Clipper applications for DOS. At that time I purchased a pretty good code generator package for Clipper / dBase (unfortunately I don't recall the name of it). The scripts that were supplied with the product were not very useful for my work, so I threw them out and wrote my own that suited the applications that I was responsible for producing.

I still remember my boss's reaction when I was able to produce complete working apps in a week while my peers were taking a month or two to accomplish the same work. And as the scripts I used matured, I was able to produce more robust applications with more features and more consistent interfaces.

The great experience I had caused me to become a huge fan of code generation.

Anyway, the day came when I began developing C and C++ applications in Windows, and because of a lack of tools, ended up resorting to hand coding everything again.
It was such a pleasure to find this book and learn about the great tools available for scripting today. I really appreciate how the author described so many instances where code generation can be applied and gave great ideas regarding how to construct generators for each case.

This is an outstanding book that I highly recommend.

Wade Matveyenkoon September 9, 2003

You Need this Book!!

I have used code generating tools in the past like javadoc, xdoclet, and ejb-gen, and have always wondered about the intricacies of code generation. Well, in Code Generation in Action, by Jack Herrington, many of my questions were answered. This is an excellent book if you are developing code generation tools, are interested in code generation, or are skeptical about code generation.

The book is split into two parts. The first part explains much of the theory and structure of code generating applications. Herrington presents the four different models for code generation and compares and contrasts them effectively. In second part of the book, Herrington goes into detail about code generating solutions. Here he presents the architectures and solutions to some common code generation problems.

I found this book to be fascinating and easy to read and understand. Herrington has been doing code generation for a number of years and has a wonderful grasp on the subject matter. The writing flows from one chapter to the next and the book continually builds on previous concepts and ideas. Another thing I liked, which others may not, was that Ruby was chosen as the implementation language for all the examples in the book. This gave me a reason to learn a new programming language, while I was learning about code generation (a 2 for 1). After reading this book, I can definitely see where I can use the ideas in my day to day development and design tasks.

Metaprogramming

In the history of computer science moving to higher level language always was the most effective method of raising programmer productivity as the latter is pretty constant in number of lines per day independently of the level of the code.  the term stress the usage of very high languages as the system that is developed in such language, even if later manually translated tend to be better structured and more maintainable then alternative approaches. 

That means that "Metaprogramming" as the method to developing computer programs stress slightly different aspect of the same technology: generating source code in a target language from a program specification in a higher level language. So this is something like a compiler form higher level language producing the program for in the lower level language.  So while is "plain vanilla" code generation we can generate language on the same level or even on higher level (disassembles and decompiles) here the levels of source and target language are fixed so the source language should be higher level language then target.

In C++ the term metaprogramming is used is a different sense: any program that generates or manipulates program code. There are several book that use this term while covering C++ template library.

Generating documentation from the program

An interesting and popular area where generative techniques are used is generation of the documentation from the program. This involves several area starting with simplest such as pretty printing or converting source in colored html (beatification), but also can involve complex methods including AI style programs for generating documentation to the code.

CGI, PHP and friends

Those methods are use to generate html "on the fly" while user is accesing the web page. 

Language oriented programming

All-in-all code generation is a very powerful programming method  which more correctly should be called "language oriented programming". This term  which probably originated by M.P. Ward in 1994 (see cse.dmu.ac.uk ) I personally like most, as in my opinion it reflects that essence of this method. This concept advocates building applications structured as a compiler or a set of compilers from several domain specific languages. As Martin Fowler noted in his article Code Generation for Dummies that, paradoxically, this approach can even be traced to creation of sed, awk, grep and several other classic Unix utilities :

Now, if we're fashionably up-to-date, we'll shun general-purpose programming languages ('GPLs') like Java or C# and do development by creating a model or writing a program in a Domain Specific Language ('DSL'). So is this new? Let's look at what happens in language terms when we use a DSL, taking regular expressions as an example.

The "regular expressions" DSL was one of the languages built in the 1970's to assist UNIX developers. The initial raison d'être for UNIX was typesetting at Bell Labs (it took a slight detour along the way) so mini-languages were created around the notion of text manipulation - sed, awk, grep and regular expression's are the ones that roll off the tongue.

A regular expression is a series of commands to find a substring of a larger string based on its shape rather than the exact text - so it was pattern matching for text. A regular expression is made up of a number of actions - basically telling the interpreter what to do to find the string...

In this sense it is amore powerful (and more realistic) alternative to dominate OO approach which advocates a single language paradigm. In a very simple, rudimentary way this is the approach used in  LAMP.  In LAMP we use several predefined languages for creating our application (typically at least HTML, PHP, Javascript and SQL). 

I suspect that all this noise about  XML as an ultimate tool for  program generation is somewhat problematic.  But idea of using XML for expressinging verys imple DSLs and, especially, as intermediary representation is not without merits... First of all because XML is readable and editable by multitude of high quality editors.  So you get some rudimentary IDE for your language from the very beginning. But that's it.  Only very simple DLS can be translated into target language using pre-existing tools for transformation of XML. As complexity of the language increases XML transformation approach loses its attractiveness.

Structuring program as a compiler

The  classic stricture of the compiler involves using lexical analyzer, syntax analyzer, code generation and a small, separate but important set of routines (and related infrastructure) for generation meaningful diagnostic messages.

Of cause not all programs can or should be structured this way. But many text processing program can and should. This approach allows producing really high quality software.

Even for applications which looks completely foreign to this approach such a Web site design there is often a way to implement part or all of the system using this approach. For example using Static content generators

Dr Nikolai Bezroukov


NEWS CONTENTS

Old News ;-)

[Aug 21, 2013] Implementing Domain-Specific Languages with Xtext and Xtend by Lorenzo Bettini

See also Xtext - Wikipedia, the free encyclopedia
Axl Mattheus on November 6, 2013

Great!! With one small caveat...

This book is extensive without being long winded. It covers most, if not all of the functionality locked away in Xtext. It does a respectable job of introducing Xtend. It is well organized, which makes it easy to skip chapters not of interest to the reader without losing context in later chapters. Examples are concise, focussed, practical and accurate - and the examples work too!

Compared to some of the works on other language construction tools, this book is a shining example of what a book on language construction should look like.

Caveat: don't take on this book if you do not have some knowledge of language construction. A little knowledge of ANTLR will go a long way as it will make a big difference in dealing with the ANTLR errors/warnings when constructing a new language.

Well done Lorenzo and the editors of this book!

[Jul 1, 2003] Code Generation in Action by Jack Herrington

This is one one the few books dedicated to this subject. and probably the most readable. See above

[Feb 7, 2001] Program Generators with XML and Java by J. Craig Cleaveland

[Cleaveland2001] From an Amazon review: "This book is definitely interesting in understanding how code generation works and how to utilize some of the newer technologies like XML and XSL to generate software."

Paperback - 448 pages Bk&Cd-Rom edition (February 7, 2001)
Prentice Hall PTR; ISBN: 0130258784 ;
Dimensions (in inches): 1.21 x 9.22 x 6.99

Avg. Customer Rating: 4.3 out of 5 stars

4 of 5 stars Worth reading if you have interest in code generation, February 21, 2002
Reviewer: A reader from San Jose, CA United States

This book is definitely interesting in understanding how code generation works and how to utilize some of the newer technologies like XML and XSL to generate software. I am very impressed with some of the new, advanced code generators like CodeCharge, which utilize XML and XSL but do not give us insight to the internals of how it works. While those tools prove that XML and XSL are great for generatng code, this book explains how it is done.

5 of 5 stars The ideas in the book are worth exploring, February 9, 2002
Reviewer: Soumen Sarkar (see more about me) from Fremont, CA United States

Agreed that XML may not be the best language to capture domain specification expressiveness. But use of XML/XSLT to do custom code generation has the benefit of rapid application prototyping and development. The crucial fact is that the domain specification is captured in XML only relatively few times and project software developers mainly use the generated code. The question is how many people in the project is exposed to 'ugliness' of XML and how many times. The advantages of 'neat' code generation far outweigh the disadvantages of 'ugliness' of domain specification in XML.

In a real Network Management Software development I achieved 60% of generated code (EJB, SNMP, Java utilities) by using custom code generation by XML/XSLT. Only myself dealt with XML other software developers happily used generated code. You can imagine the lead the project had and continues to have because of use of XML/XSLT in project specific custom code generation. The code generation system is stable now -- any new addition in EJB, SNMP model results in thousands of lines of Java/SQL/XML/SVG code without any additional effort.

I would, therefore, continue to recommend the book as worth exploring. This book really contributed new techniques in software development. More specically with XML/XSLT you have freely available tools to implement "model driven programming" in your software project.

2 of 5 stars So so, January 15, 2002
Reviewer: A reader from Victoria, Canada

While the book has interesting ideas, it ignores useful results of the domain-specific language community. More important, it preaches to use XML as a domain-specific language, which is in my opinion a disastrous idea.

Terence Parr (jGuru.com) provides an excellent argument why this is the case in his article "Answers to the question 'When shouldn't you use XML?'", August 2001, IBM developerWorks : XML zone : XML zone articles:

"XML is a poor human interface: Humans have an innate ability to apply structure to a stream of characters (sentences), therefore, adding markup symbols can only make it harder for us to read and more laborious to type. The problem is that most programmers have very little experience designing and parsing computer languages. Rather than spending the time to design and parse a human-friendly language, programmers are using the fastest path to providing a specification language and implementation: "Oh, use XML. Done." And that's OK, but I want programmers to recognize that they are providing an inferior interface when they take that easy route."

Besides, the book is poorly typeset. It appears that the font was increased until the book had more than 400 pages. I have never seen a bigger font in a computing book! I don't know why Prentice Hall endangers their good reputation with such a poorly typeset publication.

Better try to borrow the book first before potentially wasting your money.

[Jun 6, 2000] Generative Programming Methods, Tools, and Applications by Krzysztof Czarnecki & Ulrich Eisenecker

[Czarnecki_Eisenecker] From an Amazon review: "Aside from the overly academic feel to the book and touting Microsoft fantasy-ware (which may become available... who knows?) this book offers much food for thought for system designers and C++ library implementers. The template tricks described are difficult to debug but with a little luck future compilers will provide better support for this style of compile time design. I look forward to the 2nd or 3rd edition of this book when this stuff matures. "
Reviewer: childofthe80s (see more about me) from VT USA

Its hard to tell from the title of this book who will benefit from reading it but from a practical standpoint, C++ library designers and those with an interest in the "bleeding edge" of software engineering should find it very enlightening. The primary focus of this book is speeding up the lifecycle of program design by utilizing "Generative Programming". GP is a fancy name for programming using domain specific notations and generating highly optimized code without burdening the application programmer with low level details of domain libraries.

Aside from the overly academic feel to the book and touting Microsoft fantasy-ware (which may become available... who knows?) this book offers much food for thought for system designers and C++ library implementers. The template tricks described are difficult to debug but with a little luck future compilers will provide better support for this style of compile time design. I look forward to the 2nd or 3rd edition of this book when this stuff matures.

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

Top articles

Sites



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019