I had the opportunity earlier in the week to attend the 9th Symposium on Identity and Trust on the Internet (IDtrust 2010) which was held at NIST.

Given that a lot of the work that I am currently doing is centered around externalized, policy driven Authorization using Attribute Based Access Control (ABAC) and the profiling and deployment of Enterprise Attribute Services, I found a paper [PDF] and presentation [PDF] given by Ivonne Thomas from the Hasso-Plattner-Institue for IT-Systems Engineering to be very interesting.

As an aside, one of the best explanations on conveying what ABAC is all about, particularly to business owners, was given by a colleague who works for the DOD in this particular domain (Thanks Ken B).

“Consider if you will, the following two situations.

You are standing in line at the Grocery store and a little old lady in a walker comes up to you and demands your driver’s license and proof-of-insurance! You will be making a particular decision at that time. Now, consider if the same question was asked of you with red and blue lights blinking behind you and someone with a badge and a gun is knocking on your windshield asking for the same information.

We make these types of decisions all the time in our lives based on a real time evaluation of who is asking the question, what they want access to, and the context in which the question is being asked. ABAC is how we could do the same thing in the electronic world. Making a real-time access control decision based on attributes of the subject, the attributes of the resource and the attributes of the environment/context.”

I love this explanation and have shamelessly stolen and used it to great effect in multiple situations.

Coming back to the paper, given that Attributes are used to make these critical access control decisions, how does one judge the “trust-worthiness” and/or “authoritative-ness” of each attribute that are used to make the decision?  How could one convey these qualities related to attributes to a Relying Party so that it can make a nuanced access control decision?

On the authentication front, we have an existing body of work that can be leveraged such as the OMB E-Authentication Guidance M-04-04 [PDF] which defines the four Levels of Assurance (LOA) for the US Federal Government and the attendant NIST SP 800-63 [PDF] that defines the technologies that can be used to meet the requirements of M-04-04.  In particular, you have the ability to use SAML Authentication Context to convey the LOA statements in conformance with an identity assurance framework.

The paper, which I think has a misleading title, uses the Authentication Context approach as an example and defines an extension to the SAML 2.0 schema for what is termed by the Authors as an “Attribute Context” which can be applied to each Attribute value. The authors define the parts as:

• Attribute Context This data element holds the attribute context,  which is comprised of all additional information to the attribute value itself. This element is the upper container for all identity metadata.
• Attribute Data Source This data element indicates the source from which the attribute value was originally received and is part of the Attribute Context. This  can  be  for  example  another  identity  provider, some authority as a certificate authority or the user himself who entered the data.
• Verification  Context This data element holds the verification context, which comprises all information related to the verification of an identity attribute value. The Verification Context is one specific context within the Attribute Context.
• Verification Status This data element indicates the verification status of an identity attribute value, which should be one of “verified”, “not verified” or “unknown”. The verification status is part of the verification context.
• Verification Context Declaration The verification context declaration holds the verification process details.  Such a detail could for example be the method that has been used for verifying the correctness of the attribute.  Further extensions are possible and should be added here.  The verification context declaration besides the verification status make up the verification context.

I know of many folks who are working on the policy side of this question of how to judge the “authoritative-ness” of an Attribute under multiple topics such as “Attribute Assurance”, “Attribute Practice Statements”, “Authority Services” etc. etc.  But I have often thought about how one would go about conveying these types of assertions using current technology. This approach seems to provide an elegant approach for doing just that:

As you can see in the above example, the extensions proposed by the authors integrate nicely into a standard saml:AttributeStatement and convey the metadata about individual attributes to a Relying Party that can make a more nuanced access control decision.

I think this is a great beginning and would love to see the authors submit this to the OASIS Security Services (SAML) TC so that it can become part and parcel of the SAML 2.0 specification. I would also love to see a Profile come out of the OASIS SSTC that would define a consistent set of Verification Context Declarations.  In particular I believe that the concept of referencing “Governing Agreements” as defined in the current “SAML 2.0 Identity Assurance Profile, Version 1.0” (which is in public review) has applicability to this work as well.

To be conformant to SPML v2 means that the SPML interface (Provisioning Service Provider / PSP) MUST:

• Support the set of Core operations
• a discovery operation {listTargets} on the provider
• basic operations {add, lookup, modify, delete} that apply to objects on a target
• Supports basic operations for every schema entity that a target supports
• Supports modal mechanisms for asynchronous operations

There are additional “Standard” operations described in the OASIS SPML v2 Specification [Zip]. The clear thing to keep in mind is that each operations adds a data management burden onto the provider, so the choice of whether or not to implement them should be considered very carefully.

From the perspective of deployment topologies, the PSP could be deployed separately from the Target or could very well be integrated tightly with the Target e.g. an SPML compliant web service interface on a target system.

One of the frustrating items for me when enquiring about SPML support in products has been the lack of clarity and visibility around exactly what has been implemented. All too often, vendors seem to have cherry picked a chosen set of operations (whether from the Core or from the Standard list) and used that to claim SPML support. I would be very curious to see if anyone can claim full SPML v2 compliance.

A particular use case for SPML that I am currently working on has to deal with the “batch” movement of attributes from multiple systems to a central repository. The typical flow is as follows:

• Per organizational policy & relationship to user, attributes are assigned in their home organization and/or business unit (Org A / Org B / …)
• Org A must move those users and/or their attributes to a central repository (Repository X) on a regular basis
• Repository X acts as the authoritative source of attributes of users from multiple organizations / business units and can provide those attributes to authenticated and authorized entities in a real-time request/response and in a synch-take-offline-use modes.

Some points to keep in mind are:

• Org A / B / … may have, and all too often do, have their own existing identity and provisioning systems as well as associated governance processes in place.
• The organizations and the repository may or may not be under the same sphere of control and as such cannot mandate the use of the same piece of provisioning software and associated connectors on both ends of the divide.
• The systems where the organizations store the attributes of their users may not necessarily be directory based systems.
• The Repository may or may not be directory based system.
• Identity / Trust / Security are, as you may imagine, rather important in these types of transactions.

To meet these needs, we are currently profiling SPML to support the Core SPML Operations as well as the optional “BATCH” capability.  The “ASYNC” capability is something that we are more than likely going to support as well as it provides a mechanism for the provider to advertise support for asynchronous operations rather than have a request for an asynch operation fail on a requester with an error “status=’failed’” and “error=’unsupportedExecutionMode’”.

Keep in mind that the end result will satisfy more than just the one use case that I noted above. In fact, it satisfies many other use cases that we have that deal with both LACS and PACS scenarios. In addition, the profile will also bring in the pieces that are noted as out of scope in the SPML standard i.e. the Profiling of the Security protocols that are used to assure the integrity, confidentiality and trust of these exchanges. Fortunately, we can leverage some of previous work we have done in this space for that aspect.

In the physical world, when an attacker is preparing to assassinate someone or bomb a target, the first thing that they will do is to determine how best to set up that attack. The phrase used to describe the initial phase of the set-up is called 'pre-operational surveillance'. 

Unfortunately, the default configuration of most web services allow a potential attacker to do the digital equivalent of pre-operational surveillance very easily. In the digital world, these type of threats are often classified under the category of 'Information Disclosure Threats'. There are two in particular (there are more) that I would like to call attention to:

1. SOAP Fault Error Messages
2. WSDL Scanning/Foot-Printing/Enumeration

1. SOAP Fault Error Messages

All too often, detailed fault messages can provide information about the web service or the back-end resources used by that web service. In fact, one of the favorite tactic of attackers is to try to deliberately cause an exception or fault in a web service in the hope that sensitive information such as connection strings, stack traces and other information may end up in the SOAP fault. Mark O'Neill has a recent blog entry 'SOAP Faults - Too much information' in which he points to a vulnerability assessment that his company did of a bank that provided information that enabled an attacker to understand the infrastructure the bank was running and presumably allowed them to further tailor the attack.

The typical mitigation for this type of information disclosure is the implementation of the 'Exception Shielding Pattern' as noted in the Patterns & Practices Book 'Web Service Security' [Free PDF Version] which can be used to "Return only those exceptions to the client that have been sanitized or exceptions that are safe by design. Exceptions that are safe by design do not contain sensitive information in the exception message, and they do not contain a detailed stack trace, either of which might reveal sensitive information about the Web service's inner workings." (FULL DISCLOSURE:  I was an external, unpaid, technical reviewer of this book).

You can either implement this pattern in software or use a hardware device like a XML Security Gateway to implement this pattern. Mark utilized a Vordel Security GW, but this is something that can be implemented by all devices in this category. I have direct experience with Layer 7 as well as Cisco/Reactivity Gateways and happen to know that they support this functionality and I don't doubt that IBM/DataPower and others in this space support it as well.

Note that this does not imply that the error's that happen are not caught or addressed but simply that they are not propagated to an end-user.

2. WSDL Scanning/Foot-Printing/Enumeration

Appendix A of 'NIST 800-95: Guide to Secure Web Services' provides a listing of common attacks against web services, and you will note that there are many references to the information that can be found in a WSDL that can lend itself to a variety of attacks including Reconnaissance Attacks, WSDL Scanning, Schema Poisoning and more.

And in the 'Security Concepts, Challenges, and Design Considerations for Web Services Integration' article at the "Build Security In" web site sponsored by the DHS National Cyber Security Division, it notes that "An attacker may footprint a system’s data types and operations based on information stored in WSDL, since the WSDL may be published without a high degree of security. For example, in a world-readable registry, the method’s interface is exposed. WSDL is the interface to the web services. WSDL contains the message exchange pattern, types, values, methods, and parameters that are available to the service requester. An attacker may use this information to gain knowledge about the system and to craft attacks against the service directly and the system in general." 

The type of information found in a WSDL, and which can be obtained simply by appending a ?WSDL to the end of a service endpoint URL, can be an extremely useful source of info for an attacker seeking to exploit a weakness in a service, and as such should not be provided or simply turned off.

There are multiple ways of mitigating this type of an attack which include turning off the automatic ?WSDL generation at the SOAP stack application level or by the configuring the intermediary that is protecting the service end-point. For example, most XML Security Gateway's by default turn off the ability to query the ?WSDL on a service end-point.

I consider this to be a very good default.

When this option is implemented, there are often a variety of questions that come up that I would like a take a quick moment to address.

Q. If you turn off the automatic WSDL generation capabilities (i.e. ?WSDL) how are developers supposed to implement a client that invokes the web service?

There are two ways. (1) Publish the WSDL and the associates XML Schema and Policy files in an Enterprise Registry/Repository that has the appropriate Access Control Mechanisms on it so that a developer can obtain a copy of the WSDL/Schema/Policy Documents at design time. (2) Provide the WSDL/Schema/Policy files out of band (e.g. Zip File, At a protected web site) to the developer. 

Oh yes, there is always the run-time binding question that comes up here as well. What I will say is that run-time binding does not mean "run time proxy generation + dynamic UI code generation + glue code" but simply that the client side proxy and the associated UI and glue code are generated at design time, but that the end-point that the client points to may be a dynamic lookup from a UDDI compliant Registry. I've done this before and this does not require any run-time lookup of a web service's WSDL.

There is an additional benefit to this method as well. Have you ever gone through the process of defining a WSDL and Schema using best practices for web services interoperability, implemented a service using that WSDL and Schema, and then looked at the auto-generated WSDL? You may be surprised to find that the automatic generated WSDL may be in a majority of cases is not as clean or easy to follow and in some cases may indeed be wrong. The best practice for developing interoperable web services recommends following a contract-first approach. This requires that the "contract" i.e. the WSDL and the Schema to be something that is developed with a great deal of care given to interoperability. Since the automatic generation of WSDL is platform-specific, there is always the possibility of some platform-specific artifacts ending up in the contract documents, which is not what you intended to happen.

Q. What about those existing/legacy services that do a run time lookup? Won't those break?

The question that needs to be asked at this point is why these services are doing a run time lookup, is there value being added by this capability in this client, and are there alternatives that will enable the client to provide the same functionality without compromising security? 

As an example take the case of a BEA Weblogic client.  If you will look at the documentation that BEA provides on building a Dynamic client you will note that they provide two different approaches, one that uses a dynamic WSDL lookup and another that does not. The interesting thing about this is that the approach that uses the WSDL makes a run-time lookup of a Web Service's WSDL which will end up breaking if the ?WSDL functionality is turned off. But the alternative approach of building a dynamic client provides the same functionality without the run-time WSDL lookup.

From what I can see, from a functional perspective there is no difference between the two approaches and given that one of the things that you want to do when developing web services, or any software for that matter, is to minimize the number of external dependencies, I would choose the second option of NOT doing a run-time WSDL lookup in this particular case. What is regrettable in this case is that it appears that the default configuration in BEA's tooling is to use the run-time WSDL option (Or so I have been informed), which leads to issues when folks who choose the default options with their tools develop the clients. 

Mitigating these information disclosure threats requires both developers and operational support folks to understand their shared responsibility for security. Developer's need to understand that security should be part of the software development lifecycle and is not something that is bolted on at the end or is 'thrown over the wall' for someone else to take care of. Operational folks need to understand that a layered defense in depth strategy is needed and that secure coding practices of developers are an essential component of any operational environment. In particular the mentality of "Firewalls and SSL will save us all" needs to change for all parties concerned.

Notes from an on-going online discussion to self, for use as a reference and for discussion:

"SOA is an architectural style, and an architectural style is a set of principles. Gartner has enumerated five principles that constrain SOA:

• modular
• distributable
• described
• sharable
• loosely coupled

To the degree a system exhibits all five, the more it qualifies as representing the SOA style"

- Nick Gall, Gartner

"SOA Principles of Service Design:

• Service Contracts
• Service Coupling
• Service Abstraction
• Service Reusability
• Service Autonomy
• Service Statelessness
• Service Discoverability
• Service Composability"

- Thomas Erl, SOA Principles of Service Design

"From my perspective, the overarching principle governing SOA is separation of concerns. This principle helps you determine how to factor functionality into services. Thomas Erl discusses service factoring and granularity in the SOA Fundamentals section of his book rather than treating SoC as a principle"

- Anne Thomas Manes, Burton Group

"The 4 tenets of Indigo as defined by Don Box, which has now been morphed into the Microsoft tenets of SOA:

• Boundaries are explicit
• Services are autonomous
• Services share schema and contract, not class
• Service compatibility is determined based on policy"

- Don Box, A Guide to Developing and Running Connected Systems with Indigo

"The 10 Principles of SOA, as expanded on the above 4 tenets, by Stefan Tilkov:

• Explicit boundaries
• Shared contract and schema, not class
• Policy-driven
• Autonomous
• Wire formats, not programming language APIs
• Document-oriented
• Loosely coupled
• Standards-compliant
• Vendor-independent
• Metadata-driven"

- Stefan Tilkov, innoQ

I've been using a combination of Anne's separation of concerns, Thomas Erl's principles and selected bits from the OASIS SOA-RM in the SOA class that I teach but the variations above look to be great fodder for some discussions!

The W3C XML Schema Patterns for Databinding Working Group have published updated Working Drafts for "Basic XML Schema Patterns for Databinding Version 1.0" and "Advanced XML Schema Patterns for Databinding Version 1.0"

As I've mentioned before this is one of the only efforts who are looking at ways of working around the fact that XML Schema is inconsistently implemented across various web service Toolkits.

This is also the primary reason that I am moving away from databinding in general to consuming the XML directly which enables me to use native XML technologies such as XPath, XSLT etc.

What is even more interesting is that as part of the report they have come up with a Test Suite and have run it against a range of web service implementations (Axis, Axis2, .NET 2.0, WCF, XFire and more..) and documented it in an Interoperability Report. The results are interesting... to say the least!

Highly recommended reading if interested in web services interoperability.

• RELAX NG Specification at OASIS
• RELAX NG Tutorial at OASIS
• 10 reasons to model XML with RELAX NG , not W3C XML Schema
• Should you choose RELAX now?

When designing schemas, one tries to strive for modularity which allows one to build XML schemas that are composed of other schema documents. The keywords that make it possible are include, import and redefine.  Most folks who are used to schemas are familiar with import and if you want to maximize interoperability you should stay away from redefine since it is not implemented on a consistent basis.

That leaves the include keyword.  When you use include, one of the following should be true:

• Both schema documents (The including schema and the included schema) must have the same target namespace
• Neither of the schema documents should have a target namespace
• The including schema document has a target namespace, and the included schema does not.

In the last case, all components of the included schema document take on the namespace of the including schema document. The included schema document is sometimes referred to as a chameleon schema as its namespace changes depending on where it is included.

A best practice that I normally follow is to use chameleon schemas for common, reusable types so that I don't have to namespace qualify some very common schema types that I normally end up using across multiple schemas. 

I recently ran into an issue when actually working on this in that a particular .NET tool that I was using did not seem understand the use of option (3) i.e. chameleon schemas. Since I know the guys who developed the tool, and they are considered pretty much experts in the field, I was not that surprised when I pinged them on this and got back an answer that it was a known issue.

According to them, the reason that the issue exists is because of a lack of support in the .NET API itself and (this is way too low level for me) has to do with how the ServiceDescriptionImporter(SDI) class is not working properly. So you would have issues if you tried to use wsdl.exe with chameleon schemas in .NET 1.1 and 2.0. Not sure if the issue exists under WCF.

The workaround for this, which I implemented, was to qualify the included schema with the same namespace as the including schema. Not ideal, but got me to where I needed to.

Hopefully this is an issue that will be fixed.

I had the opportunity today to give a presentation on SOA and its relationship to Net-Centricity to various folks in my organization. During the Q&A session that followed the briefing, there was a question regarding service versioning.

Just to provide some context, in my briefing one of the items that I touched on is the concept of Loose Coupling and how that enables the abstraction of interface from implementation and gives a service provider the ability to change out their implementation without affecting the service consumer.

To paraphrase the question "I have a service that is being used by multiple parties, and I am changing the implementation of that service but not the interface. (1) From a testing and certification perspective, what should I do? (2) What mechanisms exist to communicate this change to all the folks who are using my service?"

The interesting variation that this particular question posed was that changing out the implementation in this example was NOT about changing implementation technology but changing the processing algorithms/business logic associated with the implementation.

My answer to (1) was that if the algorithm/business logic change had the effect of changing the expected result (as compared to the original implementation), at that point I would consider this implementation to be a whole new service and would consciously break the interface. For example, in the case of a web service implementation, I would change/update the namespace of the schema such the it would break compatibility with existing service consumers. I would also have to have this service be tested and certified as though it was a new service.

But before I do that, I would have to notify the consumers that are using my service that I am about to make this change. Which relates to my answer to (2). AFAIK, at present there is no standardized, automated way of notifying all existing service consumers that I am about to change out my implementation. So in the current state of technology, what I would have to do would be to set up a mechanism/process as part of the original client provisioning on how I as a service provider would communicate changes and updates of importance to my service consumers.

The example I pointed to was how Google implements its AdWords API versioning strategy:

The AdWords API supports multiple recent versions of the WSDL to allow developers time to migrate to the most recent version. Once an earlier version of the WSDL has been replaced by an updated version, the older version will be supported for four months after the launch of the newer version.

During this period, the AdWords API will continue to provide developer access to and documentation support for any version dating back two months.

You can tell which version of the WSDL you are accessing based on the access URL namespace, which includes the version number. Versions are named with the letter 'v' followed by whole numbers (v5, v6, etc.).

The Release Notes summarizes changes between versions. In addition, new versions and shutdowns of older version are announced via the AdWords API Blog.

In addition to this documentation, whenever we release a new version of the AdWords API, new versions and older version shutdowns will be announced via the AdWords API Blog.

In the above example, the communication mechanism is the AdWords API Blog and it is incumbent upon service consumers to subscribe to it to keep updated on what is going on with the API. And Google provides a 4 month window in which they run both the old version and the new version side-by-side to give you time to move from one to the other.

But I have to admit that this is a situation that I have not personally run into (change in implementation logic, no change in interface), so I am basing my answers on various community best practices and conversations with folks who have had to do this.  If you have run into this particular situation before, I would be very interested in knowing how you handle this in your organization, especially any info you can share on the governance policies and processes that you have put into place to communicate upcoming changes.

One of the first things that is brought up when one talks of web services interoperability is the Web Services Interoperability Organization (WS-I) and conformance to the WS-I basic profile, and how that ensures interoperability (Allrighty, I am deliberately choosing not to talk about how WS-I punted on XML Schema profiling and how you can build web services that are WS-I basic profile compliant but NOT interoperable).

Many folks have the impression that the WS-I is a standards organization. It is important that it is not and that it is a coalition of vendors.  There is a rather interesting blog post by Erik Johnson, the former chair of the WS-I XML Schema Planning Working Group, that sheds some light on some of the internal processes at this organization.

As someone who is actively involved in the standards work that is going on at OASIS, it is always fascinating for me to get insight into how other organizations work with specifications and standards in the SOA and Web Services space.

I was reading through the research article "Adaptive QoS for Mobile Web Services through Cross-Layer Communication" in the current issue of IEEE Computer Magazine in which the authors are proposing something called WS-QoS framework, which is an approach to unify Quality of Service (QoS) for web services across transport, computing and app layers.  It is an interesting read.

Per the article, the discovery, selection and invocation process consists, at a high level, of :

1. Service Provider registers with a UDDI based registry. Each service has a unique interface key.
2. Potential Service Consumer queries an "offer broker" (This is a new entity in the mix) for services that match a specific interface key AND QoS requirements
3. The offer broker acts as the middle man in identifying the "best match" between the QoS requirements of the Service Consumer and potential Service Providers who are registered in the UDDI based registry.
4. The Service Consumer directly invokes the identified best match Service Provider.

The manner in which QoS is codified is based on a specific XML Schema .

For example, for the Transport Segment you could have something like:

<operationQoSInfo name="myOperation">
...
<transportQoSPriorities>
<delay>5</delay>
<jitter>3</jitter>
....
</transportQoSPriorities>
...
</operationQoSInfo>

 For Servers it could be:

<serverQoSMetrics>
...
<requestsPerSecond>30</requestsPerSecond>
...
</serverQoSMetrics>

And at the App Layer you could have something that codifies facets like compression and decompression and other items.

As a thought exercise, given that the point of using web services is all about interoperability, I went through what would need to happen from the standards and vendor support to make all this real.

1. Given the amount of work going on around WS-Policy, wrap up the QoS information as a domain policy language for QoS under the WS-Policy umbrella
2. The direct integration of the "offer broker" functionality into the Registry/Repository
3. Built in support from the networking vendors that can map the codified policies into the appropriate technology specific network mechanism such as priorities for expedited forwarding, assured forwarding, best-effort etc.
4. Built in support from the server OS vendors that can map the codified policies into server performance levels. And given that a lot of folks are using virtualization in their computing tier, support from those folks as well.
5. Last but not least, agreement and profiling of the specification(s) between all of the web service stack vendors.

I am sure that I have grossly over-simplified a lot of things in the above and probably gotten some of it completely wrong. But the essence remains. Beyond this being a technically challenging problem, there needs to be significant amount of agreement between a lot of vendors as well as the incorporation of a variety of these technologies into the various product sets (Vendor Politics, Oh My!).

It is going to be a while! <sigh>

There is currently a war of words going on regarding the performance of some of the Java web service stacks including Axis2, XFire and JAX-WS 2.1 FCS. 

Instinctively, I think that this type of testing is asking the wrong questions and I am trying to articulate why that is so.

To start with, these steps seem to completely sidestep any of the design considerations that are associated with the development of any serious enterprise class web service. Those design considerations [Microsoft PAG: Improving Web Services Performance] include:

• Design chunky interfaces to reduce round trips.
• Prefer message-based programming over RPC style.
• Use literal message encoding for parameter formatting.
• Prefer primitive types for Web services parameters.
• Avoid maintaining server state between calls.
• Consider input validation for costly Web methods.
• Consider your approach to caching.
• Consider approaches for bulk data transfer and attachments.
• Avoid calling local Web services.

Secondly, this type of benchmarking tends to focus people on the immediacy and synchronous nature of web services rather than designing the system for asynchronous operation.  In a real production system, all too often the chunk of time that is taken up by the processing associated with the business logic that the web service is fronting may be a significant factor in the performance overhead associated with the web service. And as you go through your perf optimization, it may very well make sense to optimize that piece e.g. database calls (as that gets you the biggest bang for the buck) as you do your end to end performance engineering.

Lastly, this all seems to be about the relative performance of the various data binding frameworks that are out there (ADB, JiBX, XMLBeans etc.. etc..) which in turn brings up all the nasty interoperability issues related to serialization/de-serialization that deal with the impendence mismatch between XML Schema and the language of your choice (Java, C#, ...).  This is something that I have had a great deal of interest in, especially in trying to find work arounds to ensure interoperability. But more and more, I am becoming frustrated by this particular aspect of web services and am moving more towards avoiding serialization entirely and processing a message directly. This would also allow me to utilize some of the more powerful capabilities like XSLT/XPath etc.

Of course, this also moves me away from the web services mainstream and the "ease of use" argument that can be made due to the tooling support for XML to Object Mappings by the various vendors.  One of the things on my list of near term to-do's is to explore how hard/easy it would be to go down this path using some of the modern web service stacks such as WCF and Axis2. I really think that in the long term, it would be much more beneficial to me to go down this path and will more than likely also help out as I try to come up to speed on REST (Another one of my to-do's).

Last Call Working Draft Review for Basic XML Schema Patterns for Databinding.  Here is some more information on this W3C Working Group:

"The W3C XML Schema Patterns for Databinding Working Group, part of the W3C Web Services Activity, has released two working drafts for review.The mission of this Working Group is to define a set of XML Schema patterns that will be efficiently implementable by the broad community who use XML databindings. Patterns which may prove useful to model include abstractions of structures common across a wide variety of programming environments, such as hash tables, vectors, and collections.

There are several ways of representing such abstracted data structures and Web Services toolkits are currently using ad hoc technologies to infer the most suitable language mapping when processing XML Schemas.Agreeing on a set of XML Schema patterns for which databinding optimizations can be made will facilitate the ability of Web services and other toolkits to expose a more comprehensible data model to the developer.

The WG has published a First Public Working Draft for "Advanced XML Schema Patterns for Databinding Version 1.0." This document defines an advanced set of example XML Schema 1.0 constructs and types in the form of concrete XPath 2.0 expressions. These patterns are known to be in widespread use and considered to be compatible with databinding implementations. Implementers of databinding tools may find these patterns useful to represent simple and common place data structures. Ensuring tools recognize at least these simple XML Schema 1.0 patterns and present them in terms most appropriate to the specific language, database or environment will provide an improved user experience when using databinding tools.

The WG has also issued a Last Call Working Draft for the "Basic XML Schema Patterns for Databinding Version 1.0" specification. A databinding tool generates a mapping between XML 1.0 documents which conform to an XML Schema 1.0 schema and an internal data representation. For example, a Web services databinding tool may use XML Schema 1.0 descriptions inside a WSDL 2.0 or WSDL 1.1 document to produce and consume XML and SOAP messages in terms of data structures in a programming language or data held inside a database."

Given that the impedance mismatch between XML Schema and Language Types are one of the major causes of Interoperability problems in web services toolkits, this work and these documents are definitely worth checking out.

LOL!  From Pete Lacey at the Burton Group:

SG: [....] From here on in we pass around coarse-grained messages—you like that term, coarse-grained?. Messages that conform to an XML Schema. We call the new style Document/Literal and the old style RPC/Encoded.

Dev: XML Schema?

SG: Oh, it’s all the rage. Next big thing. Take a look.

Dev: (Reads XML Schema spec). Saints preserve us! Alexander the Great couldn’t unravel that.

SG: Don’t worry about it. Your tools will create the schema for you. Really, its all about the tooling.

Dev: How are the tools gonna do that?

SG: Well, they will reflect on your code (if possible) and autogenerate a compliant schema.

Dev: Reflect on my code? I thought it was all about documents, not serialized objects.

SG: Didn’t you hear me? It’s all about the tools. Anyway, we can’t expect you to write XML Schema and WSDL by hand. Besides, its just plumbing. You don’t need to see it.

Dev: Whoa, back up. What was that word? Wizzdle?

SG: Oh, haven’t I mentioned WSDL? W-S-D-L. Web Services Description Language. It’s how you specify the data types, parameter lists, operation names, transport bindings, and the endpoint URI, so that client developers can access your service. Check it out.

Dev: (Reads WSDL spec). I trust that the guys who wrote this have been shot. It’s not even internally consistent. And what’s with all this HTTP GET bindings. I thought GET was undefined.

SG: Don’t worry about that. Nobody uses that. Anyway, your tools will generate a WSDL, and in the WSDL will be the schema.

Dev: But shouldn’t it be the other way ‘round? Shouldn’t I design the contract first and then generate the code?

SG: Well, yeah, I guess that sounds right in principle. But that’s not so easy to do, and very few SOAP stacks support WSDL-first development. Just let the tools worry about it.

This is so darn funny, especially when you consider that it is so true! Go read the entire thing, please!

From the announcement:

Apache Axis2 is a complete re-design and re-write of the widely used Apache Axis engine and is a more efficient, more scalable, more modular and more XML-oriented Web services framework. It is carefully designed to support the easy addition of plug-in "modules" that extend its functionality for features such as security and reliability.

Major Changes Since 1.0:
- - Significantly improved documentation
- - Significantly improved support for POJO services and clients
- - Significantly improved support for Spring services
- - Significantly improved Axis Data Binding (ADB) to increase schema coverage and overall stability
- - Improved service lifecycle model
- - Improved JMS support
- - Improved handler and module interfaces
- - Improved Eclipse and Idea plugins
- - New Attachments API for sending & receiving MTOM and SwA attachments
- - Built in support for WS-Policy via Apache Neethi
- - Added support for unwrapping Web service requests
- - Fixed tons of small and not-so-small bugs
- - Major refactoring of release structure to make usage easy

Known Issues and Limitations in 1.1 Release:
- - Unwrapping of response messages (coming in 1.2)
- - JSR 181/183 Annotation support (coming in 1.2)
- - JaxMe and JAXBRI data binding support is experimental

I have updated my "Install and configure Apache Tomcat/Axis for web service development on Windows XP SP2" post for Axis2 1.1.

There was an article a couple of days ago at SearchWebServices.com titled “SOA with J2EE and .NET: Possible, but not easy” that I took a bit of exception to given that I have a great deal of interest in web services interoperability across platforms and technologies. I responded in the article comments but am reiterating that response here as well:

“If this article had come out 5 years ago, I would have less of an issue with this. I would say that interoperability is not only possible and there are accepted best practices for how to accomplish this. Most vendors now have a pretty good interop story to tell BUT it is NOT the default choice in their Tooling! Heck, I can build non-interoperable services with both sides being Java/J2EE.

The reality is that if you follow the right practices, you can build interoperable services in any web service toolkit, but if you follow vendor's path of least resistance i.e. let the vendor's tools do the thinking for you, you will be going down a path that leads to lock-in on that vendor's way of doing things. And the right practices start with Top-Down/Contract-First style of development, WS-I compliance (which does not address schema support in toolkit issues), and an understanding of some of the mine fields in the area of XML schema design for web services.

Here are some pointers that may help:

• Schema-First/Contract-First/Top-Down Style of Development & Tools
• SOAPBuilders Discussion Summary on Web Services Interoperability Issues (March 2005)
• XML Schema to Object Mappings - Issues and work arounds... “

There is still a lot of work that needs to be done in the web services Interop arena, especially in the area of the advanced WS-* specifications. But, at the same time, let us not make the problem harder than it needs to be either!

Sanjiva has announced the release of “Tungsten” which is the first product from WSO2 and brings together an integrated stack of Apache web services technologies which include:

• Apache Axis2 (SOAP)
• Apache Axiom (High performance XML)
• Apache Rampart/Apache WSS4J (WS Security)
• Apache Sandesha2 (WS Reliable Messaging)
• WS- Addressing
• Apache Neethi (WS Policy)
• Apache XML Schema
• Apache Derby (Database)
• Hibernate (Persistence)
• Jetty (HTTP server)
• Apache Tomcat

Of course, the product is completely open source and is licensed under the Apache Software licence. It currently has support for the following WS-* specs:

• SOAP 1.1/1.2
• WSDL 1.1
• MTOM, XOP & SOAP with Attachments
• WS-Addressing
• WS-Security
• WS-SecurityPolicy
• WS-ReliableMessaging
• WS-Policy
• WS-Policy Attachment

I do have to check this out!

A working group at the W3C that is looking at addressing the impedance mismatch that occurs when trying to map XML Schema to language implementation is the “XML Schema Patterns for Databinding Working Group”. They have published a working draft [1] that should interest folks who are interested in web services Interop.

Abstract

This specification provides a set of simple XML Schema 1.0 patterns which may be used to describe XML representations of commonly used data structures. The data structures described are intended to be independent of any particular programming language or database or modelling environment.

[1] http://www.w3.org/TR/xmlschema-patterns/

[Now playing: Ankhiyan Na Maar - Ek Khiladi Ek Haseena]

I've written previously about the mismatch that sometimes happens when trying to map XML to a language implementation class.  Some of the issues that I've run into are probably worth documenting.

• In consistent support for the <choice> content group in JAX-RPC
         
  When you are defining content groups in a schema, you can use Sequence, Choice or All. We had a case where the most appropriate one to use would have been the <Choice> content group. Unfortunately JAX-RPC implementations do not support this XSD construct consistently. [Note to self: Need to find out if the latest release of Apache Axis (1.3.x) has improved support for this]
  UPDATE (12/16/05): Was informed today that the JAX-RPC implementation of Apache Axis 1.2.1 supports <choice>.
  
  The work around for this was to use a <Sequence> content group but mark the elements as optional i.e. use minOccurs=0.
  
• Lack of support for an empty instance of the anyURI data type in .NET
         
  In .NET v1.1, there is a problem with schema validation of xs:anyURI data type in the .NET System.Xml.XmlValidatingReader. In reading an explanation of this from Dare Obasanjo [MSFT] some time ago, it would appear that he schema validation engine in the .NET Framework uses the System.Uri class for parsing URIs. This class doesn't consider an empty string to be a valid URI which is why the .NET schema validation considers the empty instance of anyURI to be invalid according to its schema.
  
  The long and the short is that this is supposed to be fixed in the .NET 2.0 [Note to self: Verify!]. The work around for this is to change the data type from xs:anyURI to xs:string. 
  
• Lack of support for XSD Schema Substitution Groups in the .NET XML Serializer. 
          
  In some of the schemas that I work with, Substitution groups are used extensively. I would think that one of the work arounds for this is to use the <choice> content group. The issue of course is that we cannot because of the first issue; JAX-RPC support! So we are in effect caught between a rock and a hard place.
  
  The work around for this was to go through each of the elements that use a substitution group and make an explicit choice on which one we will use/support and modify the schema to reflect that. [Note to Self: Bug Microsoft to support this construct ]
  
• Lack of support in Java for default values and optional attributes.
  
  The optional and default attributes in a schema are simply ignored when you generate the server-side bindings. On the other hand, I really am not a fan of validation adding items that will be passed on the wire without my explicit say-so, hmm..
  
• In the Java type model, all object references can be null but in .NET 1.1 value types always have a value and as such cannot be set to null. Only reference types can be null. Nullable types in .NET 2.0 address this issue.
  
  Be aware of and avoid the usage of nillable=”true” for value types! Another recommendation that I’ve seen is to define a complex type to wrap a value type.
  

Just to close out, here is a listing of Cross-Platform Safe Types:

* avoid nillable=”true”
** avoid QName values (colons in data)

From those who have experienced the pain of working through Interop, I would appreciate any corrections, additions and work-arounds to the listing.

I am a big fan of Top-Down/Contract-First Style of web service development as a mechanism to improve web services interoperability.  I recently came across a series of articles from IBM on improving interoperability between J2EE and .NET. I wanted to explicitly call out their summary of best practices for web services interoperability:

• Design the XSD and WSDL first, and program against the schema and interface.
• If at all possible, avoid using the RPC/encoded style.
• Wrap any weakly-typed collection objects with simple arrays of concrete types as the signature for Web service methods.
• Avoid passing an array with null elements between Web services clients and servers.
• Do not expose unsigned numerical data types in Web services methods. Consider creating wrapper methods to expose and transmit the data types.
• Take care when mapping XSD types to a value type in one language and to a reference type in another. Define a complex type to wrap the value type and set the complex type to be null to indicate a null value.
• Because base URIs are not well-defined in WSDL documents, avoid using relative URI references in namespace declarations.
• To avoid conflicts resulting from different naming conventions among vendors, qualify each Web service with a unique domain name. Some tools offer custom mapping of namespaces to packages or provide refactoring of package names to resolve this problem.
• Develop a comprehensive test suite for Web Services Interoperability Organization (WS-I) conformance verification.

Here are links to the articles:

• Improve interoperability between J2EE technology and .NET, Part 1
• Improve the interoperability between J2EE and .NET, Part 2
• Improve interoperability between J2EE technology and .NET, Part 3 
• Bring J2EE and .NET together in a business process using BPEL and WebSphere Business Integration Server Foundation

Dino recently posted an entry on the usage of Date and Time values when dealing with web services interop. He also points to a W3C Note on this topic.

This entry reminded me of an earlier article by Dan Rogers [MSFT] that provides great insight into this issue regardless of platform and provides best practices for handling this particular DateTime issue if working in .NET.  A key take-away from the above article as regards to DateTime usage in web services in .NET is “When using the .NET Framework version 1.0 and 1.1, DO NOT send a DateTime value that represents UCT time thru System.XML.Serialization. This goes for Date, Time and DateTime values. For Web services and other forms of serialization to XML involving System.DateTime, always make sure that the value in the DateTime value represents current machine local time. The serializer will properly decode an XML Schema-defined DateTime value that is encoded in GMT (offset value = 0), but it will decode it to the local machine time viewpoint.”

BTW, Dan is one of the creators of XSDObjectGen which is a must have tool if working with web services on the .NET platform. If you look at the documentation for XSDObjectGen, you will note the following gem: “Special handling logic for DateTime types:  DateTime serialization properties are generated and illustrate the recommended best practice for working with local and universal time values that will properly serialize on another computer in another time zone without loss in precision.” Excellent!

Back in June of this year (June 21–22), the W3C “… held a Workshop to gather and access experiences of using the XML Schema 1.0 Recommendation on 21-22 June at the Oracle Conference Center in Redwood Shores, CA, USA. More than 30 participants attended to represent diverse communities including end users and vertical standards organizations through to vendors and the W3C XML Schema Working Group. The participants shared implementation stories and expertise and considered ways to move forward in addressing XML Schema 1.0 interoperability, errata, and clarifications.”

As has been noted before, inconsistent support for the full subset of XML Schema in vendor web service toolkits is an issue that has been brought up time an again when it comes to web services interoperability. The practical experiences of the folks who came to the table for this workshop gives one great insight into this issue. 

Here are some pointers for you to check out:

• W3C Chair’s Summary Report
• Minutes of Day 1
• The presentations by the individual submitters are great and include ones from WS-I, BT, OAGi, HL7, QuickTree, SAP, Semeiosis, Microsoft x2, IBM, Sun, Oracle, Rogue Wave and Acord and others….
• Minutes of Day 2

In addition to the presentation powerpoints linked to from the above location, there are also more text based and detailed submissions to the workshop that are available on the W3C site. For example, a PDF version of WS-I’s submission.

There are some additional resources that came out of this effort that are important going forward:

• XML Schema Wiki
• The formation of a “XML Schema Patterns for Databinding Working Group”
  “This Working Group would develop a set of patterns for common data structures of XML Schema for the purpose of simplifying the mapping of XML Schemas into programming language structures.”

In short, this is an awesome resource for folks who are into building interoperable web services.

P.S. My thanks go to Anne, who in responding to a question, pointed to some of the resources here.

As you probably noted in my last blog entry, if I am on the .NET platform, I tend to use XSDObjectGen a lot.  XSDObjectGen is a excellent tool on that .NET platform that will improve your chances of Interop with other platforms and technologies. It has now been updated to 1.4.2. Check it out.

While I was visiting Microsoft last week, I had a chance to meet Dan Rogers, who was one of two guys who actually built that particular tool. Awesome guy, with whom I spent some time talking about web services, schema design and versioning and more and who provided me with a wealth of information on those topics (Thanks Dan!). 

I did not realize it until much later into the conversation that Dan also was basically one of the inventors of UDDI and was one of the folks who spent a lot of time at OASIS hammering out the 1.0 and 2.0 specs.  He also seemed to have a lot of familiarity with the BizTalk side of the house. From what he mentioned, XSDObjectGen was something that was originally developed for some work he was doing with BizTalk, but turned out to be useful in its own right.

UPDATE:  As noted by Dan in the comments, XSDObjectGen has been updated to v1.4.2.1. This is a bug-fix release. Same link as given above.

Platform and technology agnostic exchange of messages such that they can be utilized from any language or platform. That is the promise of integration via web-services.  The contract-first style of development is a web-service development style that will improve the chance of interoperability when the technologies and platforms on both ends of the wire are different.

My environment happens to be very heterogeneous and, as such, the web-services that I develop MUST interop with other SOAP stacks.   My normal design and development style is pretty straight forward:

• Design the message schema for the web service. 
  I tend to use XMLSpy a whole lot in this phase.  The interesting thing is that there are many different ways to define XML Schemas and the design choice can seriously impact the generation of implementation classes in the technology of your choice.  There are different schema design styles such as the Russian Doll, Venetian Blind and Garden of Eden that can be followed. There are also some guidelines for XML Schema design that will tend to improve web services interoperability. I tend to be a fan of the the Garden of Eden style for web services.
• Generate a WS-I Basic Profile compliant WSDL from the message schema.
  While I occasionally do use a wrench, I am not a plumber. Hand generating WSDL is a dark and mysterious art prone to many pitfalls.  Generating WS-I Basic Profile compliant WSDL by hand and tweaking it is simply voodoo!  As such, I rely on tooling to help me in this.  My particular tool of choice here is WSCF from my friend Christian Weyer. He has versions that work from the command line, as a Visual Studio.NET plug-in, or as an Eclipse plug-in. So whether or not I am on the .NET or Java platform, I am covered. After I am done with the generation, I bring the WSDL into XMLSpy for any minor tweaking or corrections as well as to make sure that the WSDL validates.
• Generate the Service Stub
  If I am using Apache Axis, I use WSDL2Java for this purpose.  If I am on the .NET platform, I use a combination of WSCF and XSDObjectGen.  I first generate my implementation classes using XSDObjectGen by pointing them to the message schema. Then I generate the shell for the web service methods using WSCF and then hook it up to the Data Types that are generated by XSDObjectGen.  Some folks may question why I do this, when WSCF can generate the data types as well. There are a couple of answers, with the most relevant being that I simply find it easier to work with code that is generated by XSDObjectGen.
• Generate the Client Proxy
  Use WSDL2Java if using Apache Axis to generate your client proxy.  On the .NET platform, I use WSDL.exe to generate the client proxy BUT I go into the proxy class and comment out all of the data types.  I actually use XSDObjectGen to generate the data types from the XSD files and hook it up to the proxy class for the same reasons that I noted above in the Service stub generation. I do not use WSCF here for another reason, which is that it seems to enforce the Pascal casing for generated type names even if the XSD is using camel casing. I prefer to keep the casing consistent between my XSD and Code.
• Unit Tests to Verify Interop
  Trust but Verify!  Since I am building Interoperable web services, I do not want my consumers to discover issues with my services. To that end, I make sure that I write unit tests in platforms and technologies at both ends of the wire to verify interoperability. i.e. If I am writing a .NET web service and I am expecting that my consumers are going to be .NET and Apache Axis based, I make sure I write unit tests in both .NET and with Java/Axis to exercise the service.  I would do the same if I was deploying an Axis based web service.

I saw a couple of pattern implementations coming across on Microsoft Download and tracked them back to two very, very interesting articles by John Evdmon :

1) Service Patterns and Anti-Patterns

“This paper, the first of a multi-paper series, provides some fundamental principles for designing and implementing Web services. The paper starts with a brief review of SOA concepts and leads up to a detailed discussion of several patterns and anti-patterns that developers can leverage when building Web services today. Guidance in this paper is applicable to any programming language or platform for which Web services can be developed and deployed.”

Some items in this article really resonated with me such as “Avoid blurring the line between public and private data representations.” and “Service consumers must map their data representations to the schema used by the service. Some consumers may find mapping to a service's schema to be a lossy process.”, as we were just today having a discussion internally about Data Models in SOA and how it is very important to understand the differences between Service Level Data Models and Internal Data Models. Especially as to how in an SOA implemented using web services, there has to be a data translation that occurs between the Internal Data Model and the Service Level one both the Provider and the Consumer sides.

I do have issues with the assertion that if  you “…comply with the WS-I Basic Profile [it will enable the service] to be consumed by any platform or programming language that supports Web services.”  My problem with this statement is comes from painful experience gained while working with today’s SOAP toolkits to build inter-operable web services.  The reality is that WS-I improves the possibility of Interop by mandating how specific artifacts such as a WSDL should be structured, but WS-I punts when it comes to XML Schema. They basically say that the SOAP toolkit should support the approved XML Schema standard and leaves it at that.

The problem here is that no SOAP toolkit has full support for the entire XML Schema standard, so when it comes to implementation specifics there is often the possibility that an schema artifact that is supported in one toolkit may not be supported in another. An example is support for substitution groups which exists in Apache Axis but does not exist in .NET. There are examples that go in the other direction as well. Check out my earlier summary of a SOAPBuilder’s discussion if you wish to know more about this..

I do love that fact that the article advocates Top Down / WSDL First web service development (… just keep the above caveat in mind when you are designing your message schema).

The associated code for the patterns mentioned in the article can be found at:

• Document Centric Pattern
• Idempotent Pattern
• Reservation Pattern

2) Service Versioning

“This paper is the second in a multi-paper series focusing on service design principles. This paper surveys some fundamental guidelines for versioning schemas and Web services. Six specific principles for versioning Web services are identified.”

Hmm.. I need to think a bit more about this…. This is also something that is of great interest to me and is something that is being actively worked on in my space as well.. I did note that some of the Message Versioning recommendations seem to tie in very nicely with the “XML Schema best practices” document from HP.

Some time back (As in late March of this year), there was a very interesting thread discussing web services InterOp on the SOAPBuilder’s list serve. The great thing was that there were a lot of folks participating in that discussion, including folks from IBM, Microsoft, WS-I and more, and some great real life issues were brought up during that discussion. 

I had at that time made a note to myself to somehow document them, if for nothing else but for my own benefit.  I am sure the discussion thread is archived so you can check out who said what if you need to,  but here are some of the important (at least to me) points that were made during that discussion:

• Most InterOp challenges occur at the point where SOAP implementations attempt to map language objects into XML data and vice versa. A key factor affecting InterOp is the impedance mismatch between language type systems and the XML type system.
  
• The impedance mismatch in question is caused by various web service stacks implementing a subset of the W3C XML schema...... and they are not all the same subset!
  
• Worse, the toolkits implement the same subset in different ways. My favourite example is .NET 1.0's problems with handling xsi:nil for value types like integers and enums and how that interacts poorly with Axis' tendency to use xsi:nil everywhere. You can argue about which platform is wrong but in the end of the day we're still left with it being difficult to build interoperable applications.
  
• ..just to clarify -- Anil said that you shouldn't have problems if you are using the same technology/platform -- and by that he refers to a particular SOAP implementation -- not just a language. You may experience interop problems going Java-to-Java if, for example, you are using Apache Axis on one side and Sun JWSDP on the other.
  
• An example on the Java side is that the <choice> content group is not supported in JAX-RPC while an example on the .NET side would be that  Substitution groups are not supported by the .NET Xml Serializer.
  
• The idea behind the WS-I Basic Profile was to make sure that a service description defines an unabiguous wire format.  One of the first decisions taken by the WS-I was to drop SOAP encoding in favor of XML Schema to express the wire format for data types.  Further, the Basic Profile expressly allows the use of all types and constructs in the W3C XML Schema 1.0 Specification 2nd Edition.
  
  Of course, lots of toolkits have problems with XML Schema -- especially around language bindings.  But this is a thorny problem to solve.  The text of the XML Schema spec can be hard to digest and type systems are complex subjects.  Even with its shortcomings, there aren't really any major ambiguities or internal inconsistency in the XML Schema spec itself.  This is a different situation than with some of the original web service specs, which partly led to the creation of WS-I Basic Profile.
  
  So how could this problem be resolved?  You could "profile away" XML Schema features and disallow the use of constructs that cause toolkits to choke.  But many (including me) think this is a bad idea.  What would you remove and for what language/toolkit problem (since toolkit support for XML schema varies widely)?
  
  Would you eliminate the use of xs:nonPositiveInteger because there is no directly related type in most languages?  That would mean those who use XML Schema validation would have to implement that particular bounds checking in our code. 
  
  The other issue is that there are lots of people (including me) who don't want to see the capabilities of XML and Infosets reduced for the sake of easier binding to languages.  But there are also lots of people who just want object-based (for example) code to remote in an interoperable way via SOAP.
  
  So two bad outcomes of all this would be (a) winners and losers get picked or (b) multiple web service standards merge (perhaps one emphasizing full "XML capabilities", one emphasizing easy language bindings). 
  
  I'm still hopeful that the toolkits will simply continue to improve their support of the XML Schema spec.  Unsupported schema constructs are OK as long as the toolkit allows a developer to mitigate the problem.  For example, don't error out (or worse, blow up) because there's an xs:redefine present -- expose the construct as an XML node type and let the developer process it in their code.
  
• > I am curious to find out if the WS-I or anyone else currently have
  > pointers to any documentation from the various toolkit vendors that
  > show exactly which schema artifacts are currently unsupported by their
  > products.
  
  I don't know of any matrix comparing schema features and toolkit support. I wouldn't expect the WS-I to take that effort on because it wants to avoid certifying toolkits directly.  But I've heard a lot of people *wishing* for such a matrix -- in the spirit of the earlier SoapBuilders interop grids.
  
• > But therefore I wonder: given the importance that XSD bindings have
  > for interoperability, how is that the WS-I BP doesn't put any limit
  > into the XSD constructs? If they do, toolkits would agree what it
  > should be supported. In this sense, maybe the limit could be those
  > constructs that were defined in the SOAP encoding soap1.1 section 5.
  
  So which XML Schema features would you pick to eliminate?  Coming up with an XML Schema sweet spot would mean deciding which toolkits and/or languages "matter" more than others. 
  
  Also, many of the WS-* specs use XML Schema to describe message formats – not to mention that XML Schema itself has a schema.  So, in eliminating something like derivations, redefinitions, or the xs:choice compositor you can actually cause the web services stack itself to get internally inconsistent (self-non-conformant?  Hmm.).
  
  Anyway, I seriously doubt the WS-I would reconsider SOAP encoding.  It was tossed from consideration in the BP very early on and with broad consensus.  XML Schema – which wasn't a standard when the SOAP spec was initially written, BTW – was considered a superior type system for describing XML documents.  So, XML Schema was adopted even though it set the effectiveness of existing toolkits back. 
  
  I find this last bit interesting.  The WS-I is basically run by the major toolkit vendors.  And although nothing gets published without affirmation by the whole membership, the WS-I Board approves what gets voted on.  Board discussions and votes are secret and it only takes 2 "no" votes out of 9 to defeat a motion to publish material. 
  Despite all this, the WS-I published the BP even though these same vendors knew it would exacerbate areas where their toolkits were pretty weak.
  
• > although XSD is very powerful for describing type semantics, current
  > toolkits don't support all the constructs and this produces impedance
  > mismatches.
  
  The impedance mismatch is between the XSD types (hierarchies) and OO language types (rich object graphs), regardless of the abilities of the current toolkits -- the type systems are fundamentally different.
  
  I actually think that the major interoperability issues are caused from the other direction -- current toolkits aren't very good at mapping rich object graphs to hierarchical structures. Many toolkits attempt to treat SOAP/WSDL as just another distributed object system (similar to CORBA, RMI, and DCOM). The toolkits focus on generating XSD definitions from code -- and lots of developers try to expose language-specific object types (Java hashmaps, .NET Datasets, etc) through their SOAP interfaces. This approach often results in interoperability challenges.
  
  If developers start with WSDL descriptions and XSD types and generate code from them, the interop issues are definitely lessened. And if the toolkit doesn't support a specific XSD construct, the toolkit can always resort to DOM.
  
• > Can any language implement any XSD type? If the answer is not, here
  > there is an interoperability problem.
  
  I understand the argument, but respectfully disagree.  The purpose of XML Schema in WSDL is to describe the format of an XML message.  As long as that format is unambiguously described, interoperability is maintained.
  
  WSDL / XML Schema was never intended to prescribe -- or even describe -- a programming model.
  
• > If I am a Java programmer that wants to create a service that sends
  > say, a hashmap, by using WSDL-first I would define somehow a XSD type
  > (something like a list of key-value pairs), but then the toolkit will
  > not generate the hashmap type, so I would have to program the hashmap
  > behaviour myself. Isn't this an inconvenience? As a programmer, I
  > would prefer to use predefined types.
  
  Not all languages have hashmaps, and not all hashmaps are the same (e.g., C++ STL vs Java).  If you don't care about non-Java languages, then you don't need the interop provided by WSDL-first.
  
  But if you don't care about non-Java, why not just use RMI?
  
• My point of view was that by doing the WSDL-first approach the service implementation (whatever is java, .net..) cannot take advantage of the language capabilities. This means that you are going to get a map of the XSD defined types: arrays, lists, etc, but not a mapping of more complex structures like trees (unless you do the parsing yourself using DOM as it was said before). But if your language supports graph types, IMHO you can't take advantage of that using WSDL-first.
  
• Right.  And my point was that if you want to support clients written in many different languages, then this is the price you have to pay.
  
• Or, alternatively, you must build an abstraction layer between your WSDL interface and your internal object model.
  
• In our enterprise we do not consider support via DOM to be support; it is a vendor cop out that is slightly better than no support and too many vendors use DOM support to claim support.
   
  In our strictly WSDL-first-development environment, we have found the interoperability problems associated with varying xsd support to be so significant that we maintain and enforce internal standards regarding allowed/disallowed XSD constructs. Furthermore, we have found toolkit xsd support variability and implementation quality to be a significant problem and, therefore, we have standards that strictly limit the WS toolkits that may be used. We allow three toolkits and life would be improved if we could reduce it to one.
   
  In my view, the state of WS interoperability is reminiscent of CORBA interoperability. And like CORBA interoperability, these problem will be solved over time.
  
• > I'm still curious what are the existing interop issues for ints and
  > dateTimes?
  > I see it mentioned from time to time, but what are the concrete
  > examples?
  
  I know that in .NET 1.x DateTime types are "value types", which means that there is no notion of a NULL value or empty content.  So there is no way to interpret an empty node in a message to a DateTime.
  
  In .NET 2.0, all value types get a "default" value that will let NULL get interpreted at least in a consistent way.
  
  I don't know the issue with int values.  However, in .NET 1.x some integer-ish XML Schema types -- like xs:nonNegativeInteger get cast as strings in generated types.  I never got a good reason why.
  

• For ints, its the problem when you want to make them optional. So it's not strictly fair to say it's int interop, it's optional parameters that are value types in Java and/or C#.

• Right, nullable value types were not supported in .Net 1.0/1.1, fixed in 2.0 by introducing Nullable<T>.

• nil and minOccurs='0' typically get handled in different ways in different platforms, if you're lucky, the platforms you care about handle them in ways that are don't cause trouble. Unfortunately today, java tools tend to serialize nulls explicitly as xsi:nil='true' unless told otherwise, whilst .NET tends to do either that, or not serialize it at all, depending on whether its a value type or a reference type, this gets better in .NET 2.0, but that is still beta. I expect people to run into this with .NET 1.1 for quite a while. (because an xsi:nil='true' on an element that's an int in .NET will cause it to barf)
  
  >I've never personally had a dateTime problem, which in retrospect
  >surprises me. Our users have a lot of confusion about timezones, but
  >the interop is actually working the way it is supposed to.
  
  Steve Loughran has written a number of times about problems with dateTime, i've never fully understood the issue he talks about, although lots of users get confused over timezones, and whether there toolkit works with UTC or local times (as most platforms DateTime datatype typically doesn't retain TZ info).
  
  One other problem i regularly see, is that xsd:long is used in a few services but has no usable mapping to COM, COM itself does support the equivilent of an xsd:long with the VT_I8 type, but every COM environment except for C++ doesn't support the type. This for example makes it tough to use the google adword API with PocketSOAP
  
  One final problem i haven't seen mentioned, and i suspect most vendors don't even consider it an issue, is the evolution of code generated proxies. for example code written against an Axis 1.1 generated proxy is not necessarily going work when the proxy is regenerated with Axis 1.2 (as the xsd -> java type mapping rules are now different). Or with .NET if the WSDL is evolved to contain 2 instead of 1 services, .NET will start using the binding name instead of the service name for it generated classes, breaking any existing code.

• >Barring the null issue, I think the interop on basic constructs (value
  >types, structs, arrays) has been fairly satisfactory. But I could be
  >proven wrong...
  
  Between Java and .NET, I think you're right that basic interop works. And I apologize for continuing to bring up the null issues, it's just a specific example I understand really well.
  
  But the whole world isn't Java and .NET. In Perl, SOAP::Lite did a pretty rough job with document/literal up until the 0.65 release a few months ago, and even then it still fakes it in a lot of places. In Python, SOAPpy doesn't really do document/literal at all (although you can fake it talking to Axis); ZSI is better, but I don't have much experience with it. PHP now offers us three choices for SOAP but last I checked none of them really worked right. gSOAP for C is apparently quite good at talking to Axis in doc/lit.
  
  ... It's not so much that it's impossible to make doc/lit services interop, it's that it's difficult. That it requires a lot of expertise. This isn't off the shelf technology; users have to understand a lot about XML, and Schema, and they have to fiddle with the code and their usage of it. And the debugging experience for them is awfully confusing.

A very interesting and educational discussion.. Hopefully of value.

If you are into Contract-First web service development, this is definitely a must have tool. This is souped up version of the xsd.exe tool that comes with the framework.

“XSDObjectGen is similar in purpose to the XSD.exe tool that ships with Visual Studio .NET.  The key difference is that XsdObjectGen creates sample code that can be used to explore the XML serialization attributes in the .NET framework.  The sample code that is generated also demonstrates a “contract first” approach to programming that eliminates the need to focus on XML for interop and classes for internal work.  The classes generated are fully supported .NET framework code, and there are no runtime libraries or licenses required beyond the normal .NET framework licensing terms.”

I am not sure if they updated the code-base for the tool since the initial release, but the associated documentation has been significantly updated with information on Schema-First development methodologies.  In particular, the support for popular XML schema structures has been updated as part of this tool. It “….Supports (inheritance, groups, abstract types, elements, attributes, complexType, simpleType, enumeration, any, anyType).  Limited support for “interesting” schema constructs (choice, substitution groups, complex type restriction, complex type extension using regular expressions).  No support for “questionable” schema constructs (union, redefine).”

In particular, I am looking forward to exploring the support for substitution groups (It is not as seamless as I would like, but is better than the lack of support that currently exists).

Check it out!

UDDI is used extensively in my environment for Web Services Discovery (Both for design and run-time use cases). Currently Windows 2003 provides a UDDI v 2.0 compliant implementation. But earlier this year UDDI v 3.0 was approved by OASIS. The leader in the field, Systinet, already has a v 3.0 implementation out there.  In addition, they have made a commitment to move away from using proprietary schemas for describing the Taxonomies to using OWL for future versions.

Does anyone out there know when we can expect an upgrade to v 3.0 from Microsoft and if there is any move to use OWL on the Taxonomy front?

Once we have properly defined our Schema, the next step in the process is to use a platform specific tool to generate the helper classes that will map the XML Schema to Platform Code and vice-versa.  Some service-side choices on the .NET side are XSD.exe, XSDObjectGen or WSCF and on the Apache Axis side it could be WSDL2Java.  To make sure that the helper classes that are generated are indeed "helping" you, it is worthwhile exploring the various XML Schema design styles and seeing how particular choices can impact the helper class generation.

The design styles that you will most often run into have very interesting and memorable names such as the Russian Doll, Salami Slice, Venetian Blind and Garden of Eden. I am not going to attempt to provide examples of this as others have done it much more lucidly than I ever could. In particular I would point you to "Schema Design Rules for UBL... and Maybe for You" by Eve Maler [Sun] for a good overview of the above design styles.

Everything starts from the data and the only truth that exists is on the wire!

I've been doing a lot of work these days with web services that are designed to be Interoperable.  The primary driver is the existence of multiple web service stacks in my environment including Apache Axis, BEA, .NET and more.  I am not going to get into the Code-First vs. WSDL/Schema-First debate. I personally believe that you can significantly minimize the number of Interop issues if you start from the Schema rather than the Code..... provided that you start from a constrained subset of the schema that is supported by your target web service stacks.

As such, it is very important that you follow good practices when designing your schema. I've come across some resources that I use and recommend and thought that I would share them.

• XML Schema considerations for WSDL design in conformation with WS-I Basic Profile
  by Shawn X. K. Hu, Xtensible Solutions
  
• Using XML Schema's Effectively in WSDL Design
  by Chris Peltz & Justin Murray [SD West 2004]
• XML Schema Best Practices
  by David Stephenson [HP]
• W3C XML Schema Design Patterns: Avoiding Complexity
  by Dare Obasanjo [XML.com]
• XML Schemas: Best Practices
  by members of the xml-dev list

The majority of the web service work that I am involved with has platform/technology Interop in the mix. As such I lean towards the WSDL-First style of web service development. I am also pragmatic enough to try to start out with a subset of schema that is supported on both ends of the pipe.. More on that later.

Since I generate the WSDL first, I really do not want the auto generated WSDL to show up on the .NET Web service documentation page. So I actually would like the "Service Description" link to point to my manually generated WSDL instead of the one generated by "MyWebService.asmx?WSDL".

I was under the impression that setting the Location property of the WebServiceBindingAttribute would enable me to point to my WSDL. In effect, map the "Service Description" link to my WSDL location.

But unfortunately that does not seem to be happening.  Here is the relevant code:

[WebServiceAttribute(
    Namespace="urn:my:service:2005:04:16",
    Description="This is a DEV instance of myService.")]
[WebServiceBindingAttribute(
    Name="myWebService", 
    Namespace="urn:my:service:2005:04:16",
    Location="myWebService.wsdl")]

I am missing something obvious. Any ideas?

UPDATE: I got this to work ....after a fashion. The trick is to make sure that you specify the WebServiceAttribute Binding Name (i.e. "myWebService") as the Binding on the SoapDocumentMethodAttribute for EVERY SINGLE web method.

[WebMethodAttribute(
   Description="Not Implemented")]
[System.Web.Services.Protocols.SoapDocumentMethodAttribute(
   Action="urn:my:services:2005:04:16:setSomething",
   Binding="myWebService",
   Use=System.Web.Services.Description.SoapBindingUse.Literal,
   ParameterStyle=System.Web.Services.Protocols.SoapParameterStyle.Bare)]
public string setSomething(SomeType setSomething)
{
}

When you do this, the Service Description link on the documentation page still points to the auto-generated WSDL location, but instead of the complete WSDL, what you will get is a WSDL import statement that looks like this:

<wsdl:import namespace="urn:my:service:2005:04:16" location="myWebService.wsdl" />

which points to my manually generated WSDL. Very nice!

The only issue that I am running into is that one of my Web Methods is simply NOT showing up at all on the documentation page.. <sigh>. I think it has something to do with the Request Type that I am passing in.

I've been looking at information on creating interoperable web services as well as tools, and came across these resources.

Articles & Online Books

• Building Interoperable Web Services: WS-I Basic Profile 1.0
• Interoperability Notes on Apache Axis 1.1 and Microsoft .NET Framework 1.0/1.1 FAQ
• Application Interoperability: Microsoft .NET and J2EE 
• Web Services Enhancements 1.0 and Java Interoperability, Part 1
  See how the WS-Security specification can be used to securely sign Web service calls between Microsoft .NET and Java using the IBM Web Services ToolKit.  
• Web Services Enhancements 1.0 and Java Interoperability, Part 2
  In part 2, continue to explore how the WS-Security specification and implementation can be used to securely sign Web service calls between Microsoft .NET and Java.

• WS-I Implementation Tools
  "..a targeted set of tools to help practitioners verify that their Web services are compliant with industry standards and implementation guidelines."
• GotDotNet Web Service Tools
  - r2d tool
  - DeDoc tool
  - WebServiceStudio 2.0
  - Direct Internet Message Encapsulation (DIME) Validator
  - WSDL Verification tool
  - WSDL Browser tool
• YATT (Yet Another Trace Tool)
  "YATT is a project to replace the current proliferation of trace tools ( tcpTrace, proxyTrace, pcapTrace ), with a single extensible tracing tool. YATT features a new GUI built with WTL, complete with a Hex View mode, and currently ships with 2 Trace providers, one based on WinPCAP and one based on the W2K Raw sockets support. "
• SoapScope 3.0
  "A toolkit-independent Web services diagnostic system for examining, debugging, testing and tuning Web services."
• OmniOpera & Omniopera Viewer Plug-in for Internet Explorer
  "Omniopera is a WSDL and XML schema editor. The Omniopera Viewer is a free plugin for Internet Explorer® that provides an easy to understand summary of WSDL and XML Schema documents. Check WSDL schema validity and verify that all QNames resolve. Hyperlinks allow you to drill in for details, all the way back to the W3C schema-for-schemas if you need to. "
• WsContractFirst
  A Visual Studio .NET Add-In for improved 'Contract First' Web services development. Ever wanted to simply right-click on a WSDL file in Visual Studio .NET and generate code from that Web service contract? Now you can - whether it be a client-side proxy class or a server-side stub skeleton: you choose. The Add-In automatically determines the project's programming language and accordingly generates source code (currently C# and VB.NET supported).

Jeff Schoolcraft made a comment on Andrew's weblog entry that:

"I must say I am somewhat disappointed that there were no examples on how cookie or session hijacking happens. It was professed to be a terrible thing, which it [hijacking] probably is, however, I would have loved to see an example of this. SQL Injection any shmuck can type ' OR 1=1 -- but what skillset, level of effort does it take to hijack a session?"

To answer your questions:

1. Cookie and Session hijacking were indeed demonstrated in the 2nd Session, when Dwayne used a Cross-Site Scripting attack to post the Cookie information and the Session Id info from a Search site onto another site. And I in my demo's showed you possible counters to this attack. What was NOT demonstrated was a cookie replay attack, which I believe was not done in the interest of time more than anything else.
   
   Note that the “session hijacking” I refer to here is getting access to the ASP.NET Session Id and doing nasty, evil things with it. I am not referring to TCP Session hijacking, which referes to an attack used by a cracker to take over a TCP session between two machines. The defense against this type of attack is more on the Admin side rather than the Dev side.
   
   
2. As to SQL Injection, it does not take much to type in the stuff for SQL Injection, but Dwayne in his session demonstrated more than that.  He showed how using SQL Injection, you can actually gather enough information on the Database schema to craft a SQL statement that allows you to taint a database. And in my session, I showed possible counters to this attack as well.