Coarray* support in gfortran
            as specified in the Fortran 2008 standard
========================================================================
* also known as CAF = Co(-)array Fortran.

Coarrays are an extension of Fortran, which allows to write parallel
programs using a Partitioned Global Address Space (PGAS) following the
SPMD (single program, multiple data) parallelization scheme. Each process
(called image) has its own private variables. Variables which have a
so-called codimension are addressable from other images. This extension
is part of the Fortran 2008 standard (ISO/IEC 1539-1:2010).

------------------------------------------------------------------------
Content                                          Last update: 2012-01-06
------------------------------------------------------------------------
- Coarray Syntax and Further Reading                      <<< What is it
- Current Implementation Status in GCC Fortran (gfortran) <<< Status
- Implementation in gfortran: Upcoming patches/plans      <<< What's next
- Examples/test files
- References for Reading
- Other Compilers with Coarray Support
- Unified Parallel C (UPC)
- Libraries for Multi-Image Support
- Committed Patched (for Reference/Historic Interest)


Contact: fortran at gcc.gnu.org ( http://gcc.gnu.org/lists.html )
         IRC: #gfortran on irc.oftc.net ( irc://irc.oftc.net/gfortran )
         http://gcc.gnu.org/wiki/GFortran

------------------------------------------------------------------------
Coarray Syntax and Further Reading                  <<<< WHAT IS IT >>>>
------------------------------------------------------------------------

The crucial ingredient are coarrays, which are variables which are
visible on all images and have a codimension. They are declared using
for instance
  CODIMENSION[1:4,5,*] :: variable
where the highest subscript needs to be a "*" (unless the variable is
allocatable - in this case, as usual, only ":" have to be used). A
remote coarray can be accessed using the bracket syntax, e.g.
  A[4] = 5
  A = 7
which writes "5" to the scalar coarray "A" on the (remote) image
corresponding to the cosubscript [4] and "7" to the local coarray
of the executing image.  Note that only a single image can be
addressed at a given time; e.g. simultaneously broadcasting to/
collecting from all images is not supported (as of Fortran 2008).

The image index can be obtained using the this_image() intrinsic,
the total number of images with num_images(). If an instruction
should only be performed on a single image, an explicit
  if(this_image() == ...) then; ...; end if
has to be used.

Besides the coarrays themselves, there exist also image control
statements such as for barriers
  SYNC ALL()
  SYNC IMAGES(<array with image indexes or "*">)
  SYNC MEMORY
or blocks which ensure that instructions are only executed on
one image at a time
  CRITICAL; ...; END CRITICAL

Note that the  STOP  statements only terminates one image; if all
images should be stopped because of an error use  ERROR STOP.


More details can be found in

- Fortran 2008 [Final Draft International Standard (FDIS)]
  ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1830.pdf

- Coarrays in the next Fortran Standard by John Reid
  ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1824.pdf

- Rational for Co-Arrays in Fortran 2008 by Aleksandar Donev
  ftp://ftp.nag.co.uk/sc22wg5/N1701-N1750/N1702.pdf

- Early draft for a Technical Report (TR) to enhance the
  coarray capabilities (teams, collective intrinsics,
  NOTIFY/QUERY image control, multi-image I/O)
  http://j3-fortran.org/doc/meeting/192/10-166.pdf
- Proposed modifictions to the TR draft:
  ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1835.txt


Historic note: Co-Array Fortran (CAF) has been developed in the 1990s by
Robert Numrich and John Reid; it has been implemented as non-standard
extension in some compilers (e.g. Cray [since version 3.1] or Rice).
When coarrays were incorporated in the Fortran 2008 standard, the term
coarrays did not not only loose its hyphen, but also some syntax was
changed (e.g. SYNC ALL is now a statement instead of a subroutine).
The last change was the renaming of ALL STOP to ERROR STOP in February
2010. Some functionality was removed when CAF were added to Fortran 2008
and has been deferred into a yet to be written technical report; this
includes collective operations.


------------------------------------------------------------------------
Current Implementation Status in GCC Fortran (gfortran) <<<< STATUS >>>>
on the GCC Trunk [4.7 (experimental)]
------------------------------------------------------------------------

GCC 4.6: Only single-image coarray support (i.e. num_images() == 1)
         but many features did not work
GCC 4.7: Also multi-image support via a communication library
         Comprehensive support for a single image, but
         most features do not yet work with num_images() > 1

For using the communication library, see
http://gcc.gnu.org/wiki/CoarrayLib -- and read the status below.


GENERAL STATUS - FOR EXCEPTIONS SEE ALSO BELOW. SUPPORTED ARE:

* -fcoarray=<none|single|lib> compile-time flag

* Declaration of coarrays

* Coarrays in expressions/coindexed expressions,
  allocatable coarrays

* Initial (but incomplete) support for polymorphic
  scalar/array coarrays

* Image-control statements:
  - ERROR STOP
  - CRITICAL block
  - LOCK and UNLOCK
  - SYNC ALL/MEMORY/IMAGES

* Intrinsics
  - num_images(), this_image()
  - this_image(coarray,...), ucobound, lcobound and image_index
  - atomic_define, atomic_ref

* ISO_FORTRAN_ENV module: The integer parameters
  atomic_int_kind, atomic_logical_kind, iostat_inquire_internal_unit,
  stat_locked, stat_locked_other_image, stat_stopped_image, and
  stat_unlocked. And the derived type lock_type.


IMPLEMENTED FOR MULTI-IMAGES (-fcoarray=lib)

Warning: There are no compile time errors and only limited run-time
diagnostic if unimplemented features are used. Use -fcoarray=lib
with num_image() > 1 with uttermost care and read the restrictions
carefully.

* For a single image (num_images() == 1) everything should work,
  which works with -fcoarray=single. This applies to both
  libcaf_single and libcaf_mpi (with "mpiexec -np 1").

* Nonlibrary-based functionality such as ucobounds/this_image
  etc. works with any number of images
* Registering and deregistering of coarrays - static and allocatable
* SYNC ALL and SYNC MEMORY
* ERROR STOP


NOT IMPLEMENTED/MAJOR ISSUES (ALSO FOR A SINGLE IMAGE)

* Still some issues with polymorphic coarrays (cf. below)
* Assignment to derived types should not reallocate
  allocatable coarray components - see below.
* Some smaller bugs and more -fcheck= run-time diagnostic


NOT IMPLEMENTED IN THE LIBRARY VERSION (-fcoarray=lib)

* Request processing loop for libcaf_mpi.
* Any communication (with very few exceptions). In particular,
  NO COMMUNICATION is performed when using:
  - lock, atomic, data push/pull
  - sync images, critical
* friendlier error stop, better run-time diagnostic


OTHER TODO ITEMS

* testsuite support for MPI
* configure/build support for the libcaf_mpi library (if useful)
  (libcaf_single.a is already automatically build and installed.)
* Improve documentation - especially move content to the .texi files
  from the two wiki pages:
  http://gcc.gnu.org/wiki/Coarray
  http://gcc.gnu.org/wiki/CoarrayLib


------------------------------------------------------------------------
Implementation in gfortran: Next steps               <<< WHAT'S NEXT >>>
------------------------------------------------------------------------

Update 6 January 2012:

TODO/in progress:

 * Polymorphic coarrays: Initial support is available, however, there
   are still some issues.

   The main issue seems to be the handling of assignments (internally)
   with -fcoarray=lib (token vs. no token), but also a similar issue exists
   for -fcoarray=single. (Tree-checking ICEs.)

   Superceded draft patch which contains a still-failing test case.
     https://userpage.physik.fu-berlin.de/~tburnus/coarray/new/select_type.diff
   [Cf. (bm) and (bn) for the commits.]
   Additional failure for scalar poly coarrays (in trans_associate_var) 

   Issues:
   - See FIXME in coarray/poly_run_1.f90: There is seemingly a
     an argument-passing issue.
   - Check PR fortran/43412 for unchecked items
   - Check for num_images > 1 issues for "dg-do run", which might have creept in.
   - CLASS: -fcoarray=lib & class container ([co]rank, w/ vs. w/o token etc.)
     but also TOKEN and select type.
   - ICE with -fcoarray=lib:
       type t
       end type t
       !class(t), allocatable :: a(3)[5:*] ! << now correctly rejected
       class(t), allocatable :: a(:)[:]
     !  allocate(a(1)[5:*]) ! OK
       call test(a)
     contains
       subroutine test(x)
        class(t):: x(3)[*]
        print *, ucobound(x)
       end
     end
   - Check that polymorphic scalar/array components are correctly handled!

* Intrinsic assignment with coarray components: No (re)alloc as the
   byte size (i.e. shape, type parameter, alloc status) has to be the
   same. See BUGS.txt
   Premature draft: http://users.physik.fu-berlin.de/~tburnus/coarray/new/caf_assign.diff
  - Check also polymorphic scalar/array coarrays.
  - Check that also deep nesting works, e.g. {a,b}%coarray%alloc_no_caf with  "a = b"

* Implement with temporary:
     sync images ([1])
                 1
Fatal Error: Sorry, only support for integer kind 4 implemented for image-set at (1)


Bug tracker and feature tracker:
  http://users.physik.fu-berlin.de/~tburnus/coarray/BUGS.txt


------------------------------------------------------------------------
Examples/test files
------------------------------------------------------------------------

As the feature is rather new, only few examples exist. (Some programs
have been adapted for the older Co-Array Fortran syntax as a search for
scientific articles shows; however, those programs do not seem to be
publicly available.)

You can find the following files in this directory:

a) ex*.f90: Smaller examples provided by John Reid.
   Some more are in the ex/ subdirectory.
b) num*.f90: Small example from a talk by Robert W. Numrich,
   http://www2.hpcl.gwu.edu/pgas09/tutorials/caf_tut.pdf
c) ca.f90: Small, Monte Carlo example by Andrew Beddall,
   http://www.fortran.gantep.edu.tr/coarrays/
d) coarray*.f90: Test files for the first attempt of creating
   a coarray patch; might be not fully correct.
e) jacobi.f90: Helmholz equation solver; adapted version based on
   http://www.co-array.org/jacobi.htm
f) andy_*.f90: Examples from Andy Vaught's coarray compendium.
g) cafClassLibrary/
   Taken from http://www.co-array.org/caf_class.htm and fixed syntax;
   especially, replaced:
   log2_images() returns the base-2 logarithm of the number of
                 images, truncated to an integer.
              -> floor(log(num_images()*1.0))
   rem_images()  returns mod(num_images(),2**log2_images()).
h) The obligatory hello_world.f90 program, which I created for
   http://en.wikipedia.org/wiki/Co-array_Fortran
i) See also the two coarray examples at
   http://www.lrz-muenchen.de/services/software/programmierung/fortran90/f03_material/2009/2009.html
   (Seemingly disappeared, but Reinhard Bader also has examples also in
    his course
j) https://sites.google.com/site/carrierpierre/computers/coarrayfortran
k) http://www.msi.umn.edu/~carrierp/coarrayFortran/index.htm


------------------------------------------------------------------------
References for Reading
------------------------------------------------------------------------

- Fortran 2008 Final Draft International Standard (FDIS)
  ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1830.pdf
  Published as as ISO/IEC 1539-1:2010

- John's Coarrays in the next Fortran Standard
  ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1824.pdf

- Toon's coarrays in gfortran
  http://ols.fedoraproject.org/GCC/Reprints-2008/moene-reprint.pdf

- Coarray rational
  ftp://ftp.nag.co.uk/sc22wg5/N1701-N1750/N1702.pdf

- Old Co-Array Fortran homepage. Not very useful as it mostly uses
  the old syntax; http://www.co-array.org/

BOOKS:
- CoArrays. Parallel Programming in Fortran
  By Robert W. Numrich, John Ker Reid
  Chapman & Hall, October 15th 2011. ISBN: 978-1-4398400-4-7
- Modern Fortran Explained
  By Michael Metcalf, John Reid, and Malcolm Cohen 
  Oxford University Press, ISBN 978-0-19-960142-4 / 978-0-19-960141-7
- Introduction to Programming with Fortran
  With Coverage of Fortran 90, 95, 2003, 2008 and 77
  By Chivers, Ian and Sleightholme, Jane
  Springer, September 2011, 978-0-85729-232-2


TR (technical report) "Enhanced Parallel Computing Facilities"
TR for things removed from the initial proposal, i.e. the
collective intrinsic subroutines, teams and features that require
teams, the NOTIFY and QUERY statements, file connected on more than
one image, except for the files preconnected to the units specified
by OUTPUT_UNIT and ERROR_UNIT. (No updated document available yet.)
See http://www.j3-fortran.org/doc/meeting/183/08-131r1.txt for the
ripe-off "patch". For the pre-ripe-off standard, see:
ftp://ftp.nag.co.uk/sc22wg5/N1701-N1750/N1705.pdf and for the
pre-ripe-of CAF intro, see
ftp://ftp.nag.co.uk/sc22wg5/N1701-N1750/N1708.pdf
Schedule: ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1812.txt

TR: FIRST NEW DRAFT: http://j3-fortran.org/doc/meeting/192/10-166.pdf
TR: Proposed modifications: ftp://ftp.nag.co.uk/sc22wg5/N1801-N1850/N1835.txt

Implementation comments by Bill Long (of Cray):
- http://gcc.gnu.org/ml/fortran/2008-04/msg00198.html (esp. item 8)
- http://gcc.gnu.org/ml/fortran/2008-04/msg00217.html

Implementation comments by Nick N. Maclaren:
-  http://gcc.gnu.org/ml/fortran/2010-03/msg00201.html
In this regard:
Cray uses according to Bill for the initialization:
"a separate 'symmetric' heap where coarrays are stored. Non-allocatable
coarrays are effectively like static arrays and they are mapped to the
beginning of the symmetric heap at program launch. There is no need to
invoke a 'collective for each local coarray', or any collective at all."
(cf. COMMON blocks). "The "symmetric" heap concept is not an absolute
requirement for coarrays, but it is an implementation convenience.  A
particular coarray is located at the same offset from the beginning of
the symmetric heap on every image.  The restrictions on allocation and
deallocation of allocatable coarrays are designed to ensure that it is
possible to maintain this address symmetry during program execution.
This makes it easy for each image to know where a coarray is stored in
memory on a different image - it has to only know the offset of the same
coarray from the beginning of the symmetric heap on the local image and
the base address of the symmetric heap on the remote image"
A potential problem according to Nick is:
"However, not all linkers produce a deterministic ordering of COMMON
blocks, especially if they are first declared in dynamic libraries"

Newer implementation comments (draft for some primitives)
by Nick N. Maclaren:
  http://gcc.gnu.org/ml/fortran/2010-04/msg00168.html

Announcement and question to steering committee:
  http://gcc.gnu.org/ml/fortran/2010-04/msg00175.html


Talks & reports ([arbitrarily] selected and unordered):
- http://usc.caseyandgary.com/sc10/tutorial/
  S10: Introduction to PGAS (UPC and CAF) and Hybrid for Multicore Programming
       Alice E. Koniges, Katherine Yelick, Rolf Rabenseifner, Reinhold Bader,
       David Eder 
  M11: Parallel Programming with Coarray Fortran
       David Henty, Alan Simpson, Harvey Richardson, Bill Long
- Fortran course slides - including "Coarray Fortran
  - A Partitioned Global Address Space Language"; second link at
  http://www.lrz.de/services/software/programmierung/fortran90/f03_material/
  Cf. http://www.rhinocerus.net/forum/lang-fortran/600690-tutorial-compiler-graphic-diagram-library.html#post241751
- HPCC 2010 Class II Submission: UPC and CoArray Fortran
  George Almási, Barnaby Dalton, Lawrence L. Hu, Albert Sidelnik,
  Ilie Gabriel Tanase, Ettore Tiotto, Xing Xue of IBM
  http://domino.research.ibm.com/library/cyberdig.nsf/papers/F7BE4A483C2295C3852577C9005A49E0
- UPC and Coarray comparison/introduction by Reinhold Bader
  http://www.prace-project.eu/documents/12_cafupc_rb.pdf
- Talks at PGAS 2009 (some headers are link to PDFs):
  http://www2.hpcl.gwu.edu/pgas09/PGAS.html
- "A New Vision for Coarray Fortran" (Oct 2009) by John Mellor-Crummey,
  Laksono Adhianto, and William Scherer III
  http://www.lanl.gov/conferences/lacss/2009/slides/Mellor-Crummey.pdf
- "Coarray Fortran: Past, Present, and Future" (July 2009) by John Mellor-Crummey
  http://cscads.rice.edu/workshops/summer09/slides/leadership-computing/CAF-Granlibakken-2009.pdf
- "A Critique of Co-array Features in Fortran 2008 Working Draft J3/07-007r3"
  by John Mellor-Crummey, Laksono Adhianto, and William Scherer
  http://www.j3-fortran.org/doc/meeting/183/08-126.pdf
- "A new vision for coarray Fortran" by J. Mellor-Crummey et al.
  http://portal.acm.org/citation.cfm?id=1809961.1809969
- "Co-Array Collectives: Refined Semantics for Co-Array Fortran"
- "Migrating a Scientific Application from MPI to Coarrays" by John V Ashby
  and John K Reid, ftp://ftp.numerical.rl.ac.uk/pub/reports/arRAL2008015.pdf
- "Introduction to Co-Array Fortran" (Aug 2009) by Robert W. Numrich!
  http://www2.hpcl.gwu.edu/pgas09/tutorials/caf_tut.pdf
- "The Complete Compendium on Cooperative Computing using Coarrays" by
  Andrew Vaught. [g95 compiler documentation] http://www.g95.org/compendium.pdf
- "A Team Object for CoArray Fortran", by Robert W. Numrich
  http://www.springerlink.com/content/k331714754381611/

------------------------------------------------------------------------
Other Compilers with Coarray Support
------------------------------------------------------------------------

- g95 (thread version or closed-source coarray library in a separate
  .tar.gz)
  http://g95.org/ & http://www.g95.org/coarray.shtml

- Cray's Fortran compiler (since 3.1, but back then using the old CAF
  syntax; price: if-you-have-to-ask-you-can't-afford-it)
  Cf. http://docs.cray.com/ -> Browse->Man pages->Fortran

- Rice's CAF (meta)compiler http://www.hipersoft.rice.edu/caf/
  CAF 1.0 seems to use the outdated syntax; CAF 2.0 is not
  yet released and will contain extensions, proposed for post-F2008.

- CAF-to-OpenMP translator -- uses outdated syntax
  http://www.co-array.org/caf2omp.htm

- Intel compiler 12.0 version:
  SMP and (cluster version) Intel MPI

- Possibly the next release of <insert your favorite major vendor here>


------------------------------------------------------------------------
Unified Parallel C (UPC)
------------------------------------------------------------------------

The PGAS equivalent to coarrays/CAF in C. There exists an UPC 
implementation for GCC's gcc, which uses GASnet (cf. below); this
implemention is named GUPC (formerly GCCUPC). While on the homepage
there is an older version, there now also exists a GCC branch based
on the trunk.

See http://gccupc.org/  (homepage)
    http://gcc.gnu.org/projects/gupc.html (description of the SVN branch)
    http://gcc.gnu.org/viewcvs/branches/gupc/ (SVN branch)
and http://gcc.gnu.org/ml/gcc/2010-04/msg00117.html (merge request)

Introduction into UPC (UPC Manual):
    http://upc.gwu.edu/downloads/Manual-1.2.pdf

GUPC implementation notes by Gary:
"Generally, GUPC "lowers" from UPC to C trees by using the
gimplifier.  Given the way that GCC has evolved, we'd be
better off if we had our own separate lowering pass, but
that requires some infrastructure that is already present
in the gimplification pass.

GUPC pre-includes gcc-upc.h, which can include gcc-upc-lib.h.
gcc-upc-lib.h can declare various runtime routines to be
inline.  Thus, in an SMP implementation a put can be mapped
essentially to a store, and a get can be mapped to a load.
GASNet uses inlining extensively.  Whenever UPC generates
calls to runtime library routines, it looks up the declaration
of the routine, and builds a call to that routine.  If the
routine is declared "static inline" then the normal mechanisms
will expand it.  This approach has pluses and minuses.
The plus is potentially more efficient runtime code.
The minus is increased complexity, and a design that
departs from the norm (for example, libcalls or __builtins).
Although allowing inline expansion of the runtime seemed
like a good idea at the time, and probably improves micro-benchmark
times by as much as 30%, I've moved to the camp that simpler
might be better."

See also: http://gcc.gnu.org/ml/fortran/2010-04/msg00098.html


------------------------------------------------------------------------
Libraries for Multi-Image Support
------------------------------------------------------------------------

There are different classes of users (a) those which just
want to have a few processes on their 4 or 8 core box under
the table and (b) those which want to use 1k or 5k processors.
(And upcoming, (c) those which want to use 1k of low-mem, slow
intercommunication GPUs.) For (a) it should run out of the box
(pthreads directly or wrapped via libgomp) while for (b) one
needs a highly optimized communication library. (Item (c) should
be postponed.)

Additionally, one has to keep in mind that hybrid programming
is common, i.e. using coarrays with pthreads, OpenMP, MPI,
shmem, GASNet, ARMCI+GA, OpenCL etc. should be possible as well.
Esp. pthreads + OpenMP or MPI with coarray <library-over-MPI>
should be taken into account, when implementing.

Note: Check with GCC@ or the GCC Steering Committee whether there
are some issues with using any of those libraries.

Note 2: Check GCC UPC implementation, which uses threads and
optionally GASnet (via Berkeley's UPC run time).

Note 3: Check GASP interface support, cf. http://gasp.hcs.ufl.edu/

Implementation: There should be a -fcoarray[=<options>] flag,
which enables non-single-image coarray support. There will be
likely several implementations in the front-end itself; however,
one likely needs a wrapper file when accessing the libraries
below (except for libgomp and possibly pthreads). One can think
of creating a simple *.o file which the user has then to link
himself; an .o file allows for LTO without GOLD linker. The
compilation should be as simple as:
  mpic -c libgfortran_coarray_mpi.c
  gfortran -fcoarray=mpi -c myFort*.f90
  mpif90 myFort*.o libgfortran_coarray_mpi.o
(If needed, one can still use "ar" to create a lib*.a to place
it into $SYSLIBDIR)

Please also check (source code, asking implementors), what
GCC UPC (GUPC) is doing - it will not only save time, but also
will make interoperability easier, which was a topic at a
recent PGAS conference.


Private suggestion: Skip thread version for the beginning and
use (c) to (e); using (d) [GASNet] might be the best (licence,
suited for task, running at HPC centres, used by GCC UPC), but
going directly for MPI (c) has also merits (extremely widely
supported, MPIv2 should also do the task, well documented and
working). Note: I had no problems to compile GASNet, but I
failed so far to get it running - I always get a single thread.


Parallel communication/data access libaries

a) libgomp (might get some non-shared memory support)
   Ships with GCC, no additional library needed
   Note: The memory model of CAF is for distributed
   memory systems - thus one needs to be careful; marking
   all variable as __THREAD and turning codimensions in
   real dimensions should work, though.

b) pthreads: Alternatively to using libgomp, one can
   access pthreads directly; might also need a wrapper library.

c) MPI, http://www.mpi-forum.org/
   Popular implementation: http://www.open-mpi.org/
   MPI (version 2.1) is very widely supported and thus highly
   optimized libraries exist. MPIv2 also supports single-sided
   communication. Maybe less optimized for "global array"
   operations as GASNet/ARMCI+GA.

d) GASNet, http://gasnet.cs.berkeley.edu/
   Used by GCC UPC [http://www.gccupc.org/]; BSD licence
   -> See test/example file in this directory

e) ARMCI (Aggregate Remote Memory Copy [Interface])/Global Array (GA)
   http://www.emsl.pnl.gov/docs/parsoft/armci/index.html
   http://www.emsl.pnl.gov/docs/global/
   Used by Rice's coarray library.
   Note: ARMCI download requires registration

[f) g95.biz's Coarray library (no source code, only few images are for free)]


------------------------------------------------------------------------
Committed Patched (for Reference/Historic Interest)
------------------------------------------------------------------------

Commit for (a) and (b):
http://gcc.gnu.org/ml/fortran/2010-04/msg00022.html
  a) Adding image-control statements
  http://gcc.gnu.org/ml/fortran/2010-02/msg00043.html
  http://users.physik.fu-berlin.de/~tburnus/coarray/new/coarray_nocodim-3.diff
  Complete single-image support for:
  - CRITICAL block
  - ALL STOP (but see (b) regarding ERROR STOP)
  - SYNC ALL/MEMORY/IMAGES
  - num_images
  b) Rename ALL STOP to ERROR STOP
  http://gcc.gnu.org/ml/fortran/2010-02/msg00178.html
  http://users.physik.fu-berlin.de/~tburnus/coarray/new/coarray_nocodim-error-stop2.diff
  Note: See http://j3-fortran.org/doc/meeting/191/10-144.txt for the renaming
  or current F2008 standard (see below).

c) ISO_FORTRAN_ENV additions
Commit: http://gcc.gnu.org/ml/fortran/2010-04/msg00027.html
  http://gcc.gnu.org/ml/fortran/2010-02/msg00055.html
  http://users.physik.fu-berlin.de/~tburnus/coarray/new/fort-env-2.diff
  Add all coarray-relevant entities to the intrinsic
  ISO_FORTRAN_ENV module, except of LOCK_TYPE.
  Note: There is a .texi typo, cf. review reply, which is fixed in
  new/fort-env-2.diff.

Commit for (d) and (e)
http://gcc.gnu.org/ml/fortran/2010-04/msg00028.html
  d) Coarray declarations
  http://gcc.gnu.org/ml/fortran/2010-02/msg00082.html
  http://users.physik.fu-berlin.de/~tburnus/coarray/new/coarray_decl-3.diff
  (Contains review fixes + PROGRAM implies SAVE)
  Coarray declaration support (no expression support!)
  Review:
  http://gcc.gnu.org/ml/fortran/2010-02/msg00137.html
  http://gcc.gnu.org/ml/fortran/2010-02/msg00139.html

  e) Coarray declaration (fixes)
  http://gcc.gnu.org/ml/fortran/2010-03/msg00032.html
  Got reviewed twice :-)
    http://gcc.gnu.org/ml/fortran/2010-03/msg00080.html
    http://gcc.gnu.org/ml/fortran/2010-03/msg00086.html
  http://users.physik.fu-berlin.de/~tburnus/coarray/new/coarray_decl_fixes.diff

g) Add flag -fcoarray=<none|single>
Commit: http://gcc.gnu.org/ml/fortran/2010-04/msg00029.html
   http://gcc.gnu.org/ml/fortran/2010-03/msg00223.html
   http://users.physik.fu-berlin.de/~tburnus/coarray/new/coarray_fcoarray.diff
   Review: http://gcc.gnu.org/ml/fortran/2010-04/msg00016.html

f) Supporting expressions except of run-time support of allocatable
   coarrays
   http://gcc.gnu.org/ml/fortran/2010-03/msg00219.html
   http://users.physik.fu-berlin.de/~tburnus/coarray/new/coarray_expr4.diff
   Works: Normal expressions work, allocate work (except for ALLOCATE
   statement), DATA works, procedure calls work,
   constraints are checked for
Reviewed: http://gcc.gnu.org/ml/fortran/2010-04/msg00055.html
Commit: http://gcc.gnu.org/ml/fortran/2010-04/msg00072.html

h) Add intrinsics (1): this_image(...), ucobound()/lcobound(), and
   image_index()
   Only for expression-constant cobounds (init expression).
   http://users.physik.fu-berlin.de/~tburnus/coarray/new/coarray_intrinsics2.diff
   http://gcc.gnu.org/ml/fortran/2010-04/msg00092.html
Additional fix (not included in *2.diff):
   http://gcc.gnu.org/ml/fortran/2010-04/msg00124.html
Commit with yet another fix:
   http://gcc.gnu.org/ml/fortran/2010-04/msg00148.html

i) Adding support in ALLOCATE for allocatable _array_ coarrays
   Submitted patch (9/n):
     http://gcc.gnu.org/ml/fortran/2010-04/msg00176.html
   Updated patch posted:
     http://gcc.gnu.org/ml/fortran/2010-04/msg00207.html
   Patch (last version):
     http://users.physik.fu-berlin.de/~tburnus/coarray/new/coarray_allocate2.diff
   Review:
     http://gcc.gnu.org/ml/fortran/2010-04/msg00286.html

j) Bound-intrinsics: Bug fix (ICE) for scalar coarrays (patch 10/n)
   Approved patch:
     http://users.physik.fu-berlin.de/~tburnus/coarray/new/coarray_intrinsics_fix.diff
   Submitted & committed:
     http://gcc.gnu.org/ml/fortran/2010-04/msg00296.html

k) Coarray array-descriptor fix (patch 11/n), PR 43931
   Committed as obvious:
     http://gcc.gnu.org/ml/fortran/2010-04/msg00324.html

l) A minor diagnostic and expr rank patch (patch 12/n)
   http://gcc.gnu.org/ml/fortran/2010-07/msg00062.html

o) First library-based multi-image support
   See http://gcc.gnu.org/ml/fortran/2011-03/msg00162.html
   Documentation: http://gcc.gnu.org/wiki/CoarrayLib

p) Extend this_image(...), ucobound()/lcobound() to support non-constant
   bounds for _allocatable_ coarrays (local + dummy) and for non-dummy
   coarrays.  (For others coarrays: See below)
   Early draft:
   http://users.physik.fu-berlin.de/~tburnus/coarray/new/coarray_intrinsics_runtime_draft5.diff
   Submitted patch:
   http://gcc.gnu.org/ml/fortran/2011-04/msg00005.html
  
   Note: ucobound assumes num_image() == 1, this_image(coarray) assumes
   this_image() == 1. Thus, -fcoarray=single is OK, =lib is not.

q) Minor coarray fix:
   http://gcc.gnu.org/ml/fortran/2011-04/msg00052.html

r) [lib] Multi-image version of ucobound
   Submitted patch: http://gcc.gnu.org/ml/fortran/2011-04/msg00041.html

s) [lib] Always build libcaf_single.a
   Submitted and committed patch at:
   http://gcc.gnu.org/ml/fortran/2011-04/msg00119.html

t) Minor ICE fix for -fcoarray=lib
   Patch: http://gcc.gnu.org/ml/fortran/2011-04/msg00164.html

u) Run-time version of IMAGE_INDEX
   Patch: http://gcc.gnu.org/ml/fortran/2011-04/msg00173.html

v) Add some missing diagnostics for IMAGE_INDEX and ALLOCATE
   Patch: http://gcc.gnu.org/ml/fortran/2011-04/msg00195.html

w) Fix maxrank check. Commited patch:
   http://gcc.gnu.org/ml/fortran/2011-04/msg00229.html

x) Small fixes for module read and allocatable coarray components
   patch: http://gcc.gnu.org/ml/fortran/2011-04/msg00244.html

y) First coarray test suite patch, using -fcoarray=single and =lib -lcaf_single
   http://gcc.gnu.org/ml/fortran/2011-04/msg00331.html
   Approved at: http://gcc.gnu.org/ml/fortran/2011-05/msg00019.html

z) THIS_IMAGE (coarray [, dim]) for -fcoarray=lib
   http://gcc.gnu.org/ml/fortran/2011-05/msg00012.html

aa) Scalar coarrays did not work with the array intrinsics,
    unless their value was known at compile time. That's
    fixed by the patch:
    http://gcc.gnu.org/ml/fortran/2011-05/msg00038.html

ab) Minor libcaf cleanup:
    http://gcc.gnu.org/ml/fortran/2011-05/msg00044.html

ac)  The following test case was failing as the
   _gfortran_caf_(this_image,num_images) variables were
   wongly declared
   Submitted patch:
     http://gcc.gnu.org/ml/fortran/2011-05/msg00023.html

  program test_image_index
    implicit none
    integer :: index1
    integer, allocatable :: a(:)[:,:,:]
    allocate(a(1)[3:3, -4:-3, 88:*])
    index1 = image_index(a, [3, -4, 88] )
    STOP ! << IMPORTANT - otherwise no link errors
    call test(1, a)
  contains
    subroutine test(n, a)
      integer :: n
      integer :: a(1)[3*n:3*n, -4*n:-3*n, 88*n:*]
      index1 = image_index(a, [3, -4, 88] )
    end subroutine test
  end program test_image_index

cd) - Save cobounds for scalar coarrays
    - Fix rank mismatch diagnostics
    http://gcc.gnu.org/ml/fortran/2011-05/msg00067.html

ae) Final(?) decl fixes for scalar coarrays
    http://gcc.gnu.org/ml/fortran/2011-05/msg00123.html 

af) Patch for coarray registering calls in the front end
    http://gcc.gnu.org/ml/fortran/2011-05/msg00176.html

ag) Some constraint checks added and minor declaration fix
    http://gcc.gnu.org/ml/gcc-patches/2011-05/msg02122.html

ah) Coarray type-decl fixes
    http://gcc.gnu.org/ml/fortran/2011-05/msg00211.html

ai) Diagnostic when passing a coarray to an allocatable noncoarray dummy
    http://gcc.gnu.org/ml/fortran/2011-05/msg00217.html

aj) Static coarrays with non-constant cobounds ("automatic cobounds")
    http://gcc.gnu.org/ml/fortran/2011-05/msg00231.html

ak) Fix cobounds of assumed-shape coarray dummies
    http://gcc.gnu.org/ml/fortran/2011-05/msg00232.html

al) ATOMIC_REF and ATOMIC_DEFINE: First, -fcoarray=single implementation
    http://gcc.gnu.org/ml/fortran/2011-05/msg00236.html
    http://gcc.gnu.org/ml/fortran/2011-05/msg00241.html

am) Parse patch for LOCK and UNLOCK (w/o lock_type support)
    http://gcc.gnu.org/ml/fortran/2011-06/msg00054.html

an) Daniel C's Fix interface (API) for SYNC ALL/IMAGES 
    http://gcc.gnu.org/ml/fortran/2011-06/msg00056.html
    http://gcc.gnu.org/ml/fortran/2011-06/msg00067.html
    Final patch:
    http://gcc.gnu.org/ml/fortran/2011-06/msg00092.html

ap) [LIB] Daniel's registering and memory-freeing patch
    http://gcc.gnu.org/ml/fortran/2011-06/msg00134.html

aq) Add LOCK_TYPE, improve LOCK/UNLOCK checking, first run-time support
    http://gcc.gnu.org/ml/fortran/2011-06/msg00125.html

ar) Add registering support for allocatable coarrays (part 1)
    http://gcc.gnu.org/ml/fortran/2011-07/msg00045.html
    http://gcc.gnu.org/ml/fortran/2011-07/msg00056.html

as) Fix registering ABI, i.e. add STAT=/ERRMSG= support and
    add error handling to the library
    http://gcc.gnu.org/ml/fortran/2011-07/msg00058.html

at) Add runtime_error function and clean up of error handling
    in the registering and in the sync functions.
    http://gcc.gnu.org/ml/fortran/2011-07/msg00087.html

au) Add runtime_error function to single.c
    http://gcc.gnu.org/ml/fortran/2011-07/msg00144.html
    http://gcc.gnu.org/ml/fortran/2011-07/msg00154.html

av) Fix runtime-error function
    http://gcc.gnu.org/ml/fortran/2011-07/msg00167.html

aw) Allocatable scalar coarrays
  http://gcc.gnu.org/ml/fortran/2011-07/msg00106.html

ax) Mark "token" (of static coarray) as restricted:
  http://gcc.gnu.org/ml/fortran/2011-07/msg00142.html

ay) Fixes to argument checking, add is_coarray function (PR 45859)
  http://gcc.gnu.org/ml/fortran/2011-07/msg00192.html

az) [fcoaray=lib] Passing the coarray token and the offset to
  call procedures [for nonallocatable coarrays]
  http://gcc.gnu.org/ml/fortran/2011-07/msg00206.html

ba) [fcoarray=lib] Pass stat/errmsg to the registering function
  http://gcc.gnu.org/ml/fortran/2011-07/msg00115.html
  http://gcc.gnu.org/ml/fortran/2011-07/msg00165.html
  http://gcc.gnu.org/ml/fortran/2011-07/msg00212.html
  LAST: http://gcc.gnu.org/ml/fortran/2011-07/msg00219.html

bb) Fix constraint check for local var decl w/ coarray components
  http://gcc.gnu.org/ml/fortran/2011-07/msg00247.html

bc) Save "token" in the descriptor
  http://gcc.gnu.org/ml/fortran/2011-07/msg00235.html
  http://gcc.gnu.org/ml/fortran/2011-07/msg00246.html

bd) Fix error handling of allocatable carrays (and other allocables):
  PR fortran/ http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49755
  http://gcc.gnu.org/ml/fortran/2011-07/msg00243.html
  http://gcc.gnu.org/ml/fortran/2011-07/msg00254.html
  http://gcc.gnu.org/ml/fortran/2011-07/msg00277.html
  http://gcc.gnu.org/ml/fortran/2011-07/msg00290.html

be) Pass token when registering allocatable coarrays
    http://gcc.gnu.org/ml/fortran/2011-08/msg00002.html

bf) Constraint fix for coarray result variables
    http://gcc.gnu.org/ml/fortran/2011-08/msg00028.html
    Post-commit fix: http://gcc.gnu.org/ml/fortran/2011-08/msg00129.html

bg) Constraint fixes for LOCK_TYPE (PR 18918)
    Cf. http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18918#c62
    http://gcc.gnu.org/ml/fortran/2011-08/msg00004.html
    http://gcc.gnu.org/ml/fortran/2011-08/msg00021.html
    http://gcc.gnu.org/ml/fortran/2011-08/threads.html#00004
    Review comment: http://gcc.gnu.org/ml/fortran/2011-08/msg00036.html
    Latest: http://gcc.gnu.org/ml/fortran/2011-08/msg00136.html

bh) Handle token and offset for assumed-shape coarrays with -fcoarray=lib
    Patch: http://gcc.gnu.org/ml/fortran/2011-08/msg00182.html

bi) Fixing allocatable scalar coarray components
    http://gcc.gnu.org/ml/fortran/2011-08/msg00210.html

bj) Deregistering of allocatable coarrays: Library part
    Patch: http://gcc.gnu.org/ml/fortran/2011-08/msg00215.html
    Updated patch: http://gcc.gnu.org/ml/fortran/2011-08/msg00238.html

bk) Initial support for scalar and array coarrays. The support is burried
    in the patch at http://gcc.gnu.org/ml/fortran/2011-12/msg00057.html

bl) this_image fixes, deallocate of scalar poly coarrays:
    http://gcc.gnu.org/ml/fortran/2011-12/msg00089.html
    This patch also fixes coarray/poly_run_2.f90 :-)

bm) Fix for cobounds intrinsics
    http://gcc.gnu.org/ml/fortran/2011-12/msg00107.html

bn) SELECT TYPE parse fixes
    http://gcc.gnu.org/ml/fortran/2011-12/

bo) Fix integer-kind issues, esp. with -fdefault-integer-8 [PR 51682]
   http://gcc.gnu.org/ml/fortran/2011-12/msg00169.html

bp) For -fcoarray=lib: DEALLOCATE support by calling caf_deregistering
    http://gcc.gnu.org/ml/fortran/2011-12/msg00157.html