2025 Update -
This was an old blog, written back when I was a Lotus Notes Dev, when SharePoint was new, and when I had a couple decades less experience.
Not all of it aged well. Not all of it was good to begin with.
Much of the formatting has gone wonky when I imported it from a Wordpress XML file. I did not import the comments. Some posts are missing, and some I removed because they were so dated or trivial that they were just noise.
Despite all that, it is a really good bed of content to test this app with. The blog did get a decent amount of attention in its day. And as I read through it all fixing paragraph breaks and such things, it was interesting to see what remains relevant 15 years later, and what is so dated that it barely makes sense anymore.
History:
I started this blog in 2007, as a 100% Notes guy, trying to learn SharePoint and figure out how to migrate Notes functions to SharePoint. The old posts cover that journey, including success, failures, and documentation.
Its original purpose was defined as:
To document the real-world situations encountered in a migration from Lotus Notes/Domino technology to a Sharepoint/.NET technology platform. I hope to be fair to both technologies, and will prefer to focus on their strengths, rather than documenting their weaknesses.
I hope to also provide some real technical answers, by writing down specific techniques used to migrate Notes functionality to the new platform. I will try to write things in a generic sense, and allow others to translate it to their own needs.
Now, I've moved on from there. I quit the job where I spent many years shutting down a Notes environment, with mixed success. I then found myself doing consulting work on Notes migrations. I now am an engineer at a Domino-driven SaaS.
So as of April, 2012, I've re-kindled the blog to have an outlet for my thoughts, and maybe even to spark some discussions that may help myself and the community in our works.
Read the following description of a technology, and tell me which one I am referring to:
The correct answer is that the above description describes both Domino and Sharepoint. It isn't a perfect description of either, to be sure. But that fact that a common description can be made of both technologies is certainly food for thought, especially when corporate politics are driving the decision of which platform to use. If the technical architecture is fairly similar, why would a corporation NOT choose the more mature product? I can think of some answers to that. But it is definitely a question worth asking.
A week or so ago, I found the following blog entry: http://rndnotes.wordpress.com/2007/09/18/why-i-like-domino/ (via Ed Brill)
The gist of it is that developers need to know more basic technologies to be a good Sharepoint developer than they do to be a good Notes Developer, therefore Notes is better. But in all fairness... that isn't fair. Both technologies have many aspects to their development. Both technologies have underlying server administration. I know almost no developers that truly know it all:
I know Notes developers who don't know anything about Notes admin. Those who do don't always know the OS level admin to suport a Domino install. Thos who know all of that might not know all the options in the notes.ini to customize server performance. And an admin who knows all that often doesn't know LotusScript. Or if he/she does, they don't know Java. Or maybe they know Java, but not JavaScript, so they are limited on the web application front.
The point is that any technology platform these days requires multiple areas of expertise. And almost nobody learns everything. The list of skills needed to run a platform need to exist at an organizational level, not within an individual. What it boils down to is that neither Notes/Domino nor Sharepoint are highly effective in one-man shops. Unless you personally are one of the few people who truly know it all (they do exist, but most people who think they know everything overrate themselves) you need a team with varied skills to run them well.
If we start to reject any technology platform that requires knowledge of multiple skill sets, the entire industry will start to stagnate. Technology is complex these days. That is why everyone in this industry specializes in something. It is a nice thought that powerful enterprise systems can be simple enough that one person can know it all, but I find that idea somewhat unrealistic. Instead, we need to seek the platforms that best meet our business goals, including the desired size and cost of our IT organizations I believe that a Notes/Domino shop can do that for most businesses. I believe Sharepoint has less flexibility, but can still meet the business goals for many businesses. But in our measurements and judgments on these platforms, I recommend that we focus on their effectiveness within our organization, not the skill sets of any individual involved in their development.
One of the first things that needs done before migrating off of Notes is to plan the migration. Sounds obvious, doesn't it? But I've now worked in three corporations that told me in the interview process that their plans were underway... yet when I started, the plan in its entirety consisted of: "Get rid of Notes sometimes in the next few years." (And the Notes people smile and don't push it because they know that unless someone is planning and directing the effort, Notes will stick around. In addition, it would be difficult for a non-Notes person to know enough details of any given Notes environment to make up a plan, so is it really any shock that migrations don't have a good track record of success to date?)
At the risk of alienation from the Notes folks, let me lay out how I would create the overall plan.
I recognize that this over-simplifies just about everything, but... I intend to expand on each of these points over the next week or two. So stay tuned. I hope to bring out some discussions as I go into more detail that will be useful to anyone else going through this process. And maybe even some code to help with analysis... maybe...
Many of you may be saying -- don't we already have an inventory via the Catalog? Partially. Notes Apps do not have to be in the catalog, for starters. But the real problem is that the catalog is not laid out in the way I would recommend. This can be resolved via design changes, but let's step back before we get into that. As one carries out this plan, the inventory acts as a checklist of action items, and becomes the tracking DB for your migration:
Look back to the previous post for a second. Each step is an action item that needs to be taken on each database. Your inventory is your tracking system for this process. It will allow you to carry out the plan in parallel on different databases, rather than sequentially on your entire environment. This in turn lets you work on the "low-hanging fruit", and not let your problematic applications (and you will have some) hold up the entire migration.
This parallel approach won't give you an overall timeline or cost for the migration, but I have yet to see a migration that plans thing out to that level of detail before attacking the easy stuff. Whether you use the catalog or make a new DB for your inventory is up to you. I prefer a new DB just because I like the simplicity of a single source of truth for my planning needs.
In either case, create fields for each of the step of the plan. Create views for each step of the plan, showing which databases need that action. I also recommend an overall status view, showing each status field, to give you "the big picture." I don't have a sample DB at the moment to illustrate this, but it isn't a very complex DB, conceptually. Do what works for you.
Now that you have the inventory DB, the trickier part is getting the data in place. It is not rocket science, just some basic scripting. My recommendation is a nightly agent that loops through every DB on the server, creating a document in the inventory for each DB found. (And updating if it already exists.) I recommend this agent gathers, at a minimum the following data for each DB:
One the DB is created and the data is in place, you are ready to move on to the next step of the plan. But -- let me point something out here. As the migration moves forward, and more and more apps are no longer in Notes, will your team really want to keep coming to Notes just for the migration tracking DB? Doubtful. There are two ways to mitigate this. The quick and easy one is to just make this a web interface and be done with it. Everything above still applies if you do this. But let's have some fun for a moment. Let's consider what it would take to make that exact same tracking DB in Sharepoint:
Assuming you already have a comfort level with .NET coding, this probably wouldn't take more than an hour or two longer than setting up the same process in Notes. If there is any interest in seeing sample code for step #5, let me know in the comments. I haven't actually written that code yet, and am short on time at the moment, but if there is enough interest, I'll sit down and crank it out for you.
I scan technorati on a daily basis for the tags of Domino & Sharepoint... I won't comment on them all, to be sure, but hey, I'm just getting started here and having fun writing. So here you go...
Today, I hit this little post: http://www.lotusguru.com/lotusguru/LGBlog.nsf/d6plinks/20071003-77MQWX The post itself is uber-minimal. As are the comments at the time of this writing. But start following the links back to the previous blog entries. It will take you back a few months to a number of discussions on Notes vs. Sharepoint. Mostly from the Notes viewpoint, but still interesting reads, if you did not see them this past summer.
For my part, I maintain that both systems serve a purpose. Notes is the more mature and flexible product, but Sharepoint integrates nicely with other MS technologies. But the question posed in the link above is really a critical question for an organization -- what DOES make Notes so good at workflow? And what makes SharePoint good? Where do they intersect? What kinds of workflow would lead you in one direction vs. the other? On second thought, is that really what matters? Do these questions help an organization decide on a technology platform, or are there larger issues at stake. Is the workflow equivalent enough that orgs will look to other factors when making decisions? Lots of questions, but I don't have answers today. I hope as I go forward learning the MS technologies, doing another migration, and writing here, I'll come up with answers over time.
This one isn't tricky, but will be critical to your success. For every Notes app, you need to identify who owns it in your business. For the apps that you support all the time, you will already know this. Some apps, IT may own directly. And you can list IT as the owner of any DB that came with the server, as it will be your call when they are no longer needed.
But there will likely be some old discussions or small apps that haven't been touched in years. Maybe the folks who used them don't even work there anymore. You STILL need to find a business owner. Someone outside of IT needs to make the decision as to whether old obsolete apps can really be deleted, or if the data needs to be migrated somewhere. Most of the time, IT will already know the answer, but there are those rare occasions that some old discussion DB held information that a department was holding on to for legal reasons, for example. You may not need to keep the app alive, but you need to work with the business on somewhere to archive the data, or something along those lines.
So let's spell out what the business owner will do with you:
I've worked in small shops where one high-level manager was willing to be the business owner for most of the apps. That makes it easy. I've also been in shops where just tracking down someone willing to take on that role was a 6 month project. Hopefully you are either small enough that this is an easy question, or large enough that you have structured change management, and therefore already have a list of approvers for any application. Those approvers are likely to either be your owners, or able to tell you who is.
(And if anyone is reading this and thinking, "How could you not know who owns an app?", believe me -- I've yet to see a Notes shop that has been around for more than 10 years in which every app is accounted for. Notes shops are often a little loose on the process side of things -- while that may breed some disorganization, it seems to work well that way.)
I was working on a view last night, and ran into a question that illustrates some of the differences in thinking between Notes and SharePoint:
Q: How you do create a dropdown list that show user-friendly words, but sorts and computes on a numeric value?
A: In Notes: Create a drop-down field, and in the choices list, separate the front-end value and the back-end value with a pipe. For example: High|1; Medium|2; Low|3. The Domino server will handle the "translation" of the words to the field in a form. You need to handle it yourself in views or other output.
A: In Sharepoint: Create a second column, computed on your words, and use that column for your sorting and/or calculations. Managing which value is used becomes your responsibility in all cases. Also, to compute a value in SP, you do have an equivalent to @Functions - it uses the syntax from Excel for calculations. I had to look up how to do nested IFs in Excel, but it did work:
IF(Priority="High",1,IF(Priority="Medium",2,3)))
(I found a reference online that stated IFs could only be nested 7 deep in Excel. Does anyone know if that is still true, and if it is also true in SP?)
A few other differences in SharePoint:
Sorry for the horrendous subject line, but this one is going to take a number of posts to get through. I'm just going to cover the easy stuff today.
The idea at this point is to define, at a high level, a solution for each app. This will not only help you have an idea of what work needs done, but the process of deciding will force you to start the learning curve of the Microsoft Technologies. We've already established that SharePoint is really just the front-end and some basic forms. We're going to need to add on other tools. This is your opportunity to identify what tools you want in you organization, and learn about your choices.
“Convert to Sharepoint” We already went through identification of all DBs that can function in Sharepoint with the out-of-the-box functions. Tag those to be converted directly. I made it a separate step in the plan instead of covering it here because I find it makes you really think about what SharePoint can do as you do a first pass through your organization with that goal in mind. SP won't do everything for you, but it is quick and easy, and therefore a noble goal if you must migrate off Notes.
“Mail-in DB: Move to Exchange” Check out your mail-in databases. Odds are, at least some of them have no workflow -- they are truly just a shared mailbox. Get those things out of the Notes system. Move them to Exchange with the rest of your email. This isn't a technical issue, it is a maintenance issue. Having dual mail systems is just a pain. So move everything over.
“InfoPath Forms" When I wrote this option, I had thought InfoPath was a robust forms/workflow solution on the MS side. But as I've learned a bit over the past two weeks, I'm beginning to see that just about every solution will require some custom code. So I'm now questioning this option. I'll leave it in the list, but my current suspicion is that a strong dev organization doesn't need this. It is another layer of abstraction over the code, which may be good if you are the kind of shop that outsources everything, but want to keep forms in-house. You could still mark basic form/workflow apps as InfoPath, but I'd keep an open mind as you go forward as to whether this is really a valid option for your organization, or if you just want to sit down and crank out some code. And if anyone out there is an InfoPath expert, I'd love to hear some info on why InfoPath is a better solution than custom code.
Look at that -- 415 words, and I didn't really say a thing. Sorry, tech folks. This one was more just pontificating on my internal thought process than anything tangible. Tomorrow evening's post should have a bit more weight to it. Check back then...
“.NET Custom Development” Code geeks, this is where we get to have fun. You can literally do anything with SharePoint as a front-end by doing custom development in .NET. When Microsoft proponents say that SharePoint is vastly powerful, this is what they are talking about. But let me explain exactly what your options are within .NET: You can write web parts in .NET. You can write pages and forms in .NET, you can customize your existing SharePoint lists in .NET. You can write huge, massive .NET apps and bring them into SharePoint. Hey, its .NET. A full programming environment. You can do it all. And THAT is where Microsoft gets away with saying SharePoint is powerful.
While I agree - this gives you all the power in the world, let us also be clear -- you're no longer working in SharePoint. You are working in ASP.NET, and using SharePoint as a front-end. And it feels like a hack to me -- to customize a SharePoint list's entry form, you open the file system and edit the aspx page. Sure, it works, but it is a hack.
But let me focus on that first part -- it works. Quite well. ASP.NET has a lot of power. It is in ASP.NET that Notes people will see the event-driven model they are used to on a form. This is where the full .NET object model is available to you, in addition to the SharePoint APIs, and yes, even the Notes COM objects. Code up some web services around your enterprise, and you can do some VERY slick integration work at this level. You can code up your forms, write code to handle every possible user interaction with every possible component, and basically start to approach what Notes gives you. You can code console apps and schedule them in the OS, and achieve the equivalent functions of your scheduled agents. (Although, let's talk maintenance issues some day.)
When you hook SharePoint on to the front to handle UI and Navigation for you, this is when it starts to be a real competitor to Notes.
Let me re-iterate that sentiment, as I think it is one the Notes people need to recognize. SharePoint alone is no match for Notes. But when you throw in .NET coding behind it, it is very competitive with Notes. Even price-wise as the .NET Framework and SharePoint Services are free. (Fancy tools like MOSS, and IDEs do cost money, though.) At this level, .NET can do just about anything Notes can. I am not so sure about field-level encryption or replication, but short of those items? I have confidence in the .NET platform for business application development. So with those 2 items in mind that SP + .NET are not so good at -- let's get back to the concept of planning. Of the apps you have remaining, how many of them do not need replication or field level security? Before you say "all our folks replicate their data locally", think about replication for a minute. Replication is much more than offline access to data. It is the algorithms to incorporate that data back into a DB. We can use XML to grant access to .NET data offline. We can even write a simple front-end for the users to use as an admin/edit interface into that data. And writing some basic code to handle updating the source DB with the offline data isn't rocket science either. What IS difficult is handling replication conflicts. So let me ask the question again a little differently -- which of your apps needs replication conflict handling after offline data processing? I suspect when you stop to really think about this, the number will be greatly decreased.
Remember, our goal here is not to sell Domino. It is to figure out how to meet the business goals without Domino, and identify what the true technical issues are. So where do we now stand? We've identified which apps can be placed into SharePoint, and which ones need custom .NET coding. We've also probably identified a few that do need some specific Notes features. Let's hold onto those until we finish the last few options. My next post will cover our last three categorization options, and then we'll move on to some planning steps that will take some technical analysis and scoping. I know I'm talking at a very high level right now, but we'll get deeper as time goes on... I promise...
A few more notes on the differences between a drop-down list in SharePoint vs. Notes.
1) In SharePoint, when you select a field to list choices, it assumes you are going to hard-code the list. If you want a dynamic list of choices, it is a different data type when setting up the column --the lookup type. At first glance, this seem like no big deal - just make your selection for what you want. But - what happens in 6 months when the list changes, if you hard-coded it. That first time you are asked to change the list, you will say to yourself, "Ah, I should make this a lookup." Now you have to go write code to update all the existing data to support that change. This is nowhere near as easy as Domino, where you simply change the list to a formula.
2) As a symptom of the above problem, you need to create your source data lists to support your lookup lists before you create the form. This to me seems very unintuitive. I tend to get requirements from the end users, prototype a form with basic fields, then go back in and build the supporting lookup documents. Because both Notes and SharePoint are rapid, document-driven platforms, I am surprised that SharePoint is coded in such as way that it punishes you for quick prototyping of a form before all the details are put together. You can certainly still make up a form, and make changes later, it just becomes more tedious. And as mentioned above, if you make changes after data exists, you may end up writing code to fix the data. (And agents don't exist... that code must be custom .NET apps. Even for basic data updates.)
3) How many people remember the days of Notes R2/3, when almost every new developer asked the question of how to make a drop-down list generate itself based on the already existing data in the app? When a user entered a new item in the list, the next user got that new item in their drop-downs. In later versions, as we added more features, this wasn't the standard newbie question anymore, but it does illustrate a good concept: How would one answer that question in SharePoint? I may have to try setting a lookup field in a document to its own list, and see if it works. If it is that simple, great -- because trying to dynamically update a secondary list based on a new data value in a document doesn't seem like it would work very well.
Based on the stats, you guys care a lot less about planning, and a lot more about technical details.
So be it -- one last short post to finish out the plan, then we'll get more detailed posts coming.
If everything is now scoped out, you have everything you need to prioritize your migration. Prioritization is going to be based on your business needs, naturally. But let me make just a couple suggestions:
1) Put the apps that require the most maintenance near the front of the list. That will give you an opportunity to correct the problems causing maintenance, as well as leave you with a low-maintenance Notes environment, so you can spend your time on more valuable tasks
2) Put client-based apps near the front of the list, too. I've mentioned before that simply getting to a 100% web-based environment should offer maintenance and cost benefits to your organization.
Then, the final step - do the migration. I'm not even going to attempt to comment on that. A lot of work to be done.
From here on out, I'll try to continue to post technical issues and techniques. That is where all the fun is to be had anyway. :)
One big gap in SharePoint's functionality is a lack of agents. The concept just doesn't exist. So I wanted to take a brief look at what we really did with agents, and how you could achieve the same results with Microsoft technologies.
I'm not posting how-to instructions here, just a general direction that can be taken. It should at least be enough to help you know what to search for on Google.
1) Data Updates -- If you need to make mass data updates, (or even just small corrections to specific documents), a previous commenter pointed out that one of your best options is actually Microsoft Access. SharePoint can be a data source, and you can use Access to update your data.
2) Event-driven scripting -- We're used to our plethora of events -- we can and do script to the QuerySave, the PostSave, the POstOpen, the QueryDocumentDelete, etc. We can control just about everything in our system based on an event-driven model. But SharePoint doesn't have this. ASP.NET does.... somewhat. To take advantage of an event-driven model, you would need to write an ASP.NET application, and either turn it into a Web Part, or just link it from SharePoint.
3) Scheduled Agents -- You will need to write a console app and schedule it at the Operating System level.
4) Scripted Actions -- While SharePoint doesn't have the action bar we are used to, it does have menus. And you can update those menus by writing and deploying custom features. What it comes down to is that data updates aren't as difficult as I first feared, and everything else is custom .NET coding. While it is not integrated with SharePoint the way we Notes folks would like, it will serve the same purpose. And who knows? Maybe if enough of us learn the Microsoft side of things, we can start to explain to them what an integrated programming environment actually is. :)
I mentioned a while back that you could keep track of your Notes DBs in SharePoint. Below is a bare-bones snippet of how you might set that up. I wrote and tested this as a Windows console app.
You will need to set up a project in VS.NET, and add references to the SharePoint API and the Domino Objects via COM. In a real-world scenario, this would need major improvements. Namely, either splitting the list generation into a new script, or checking if it already exists. Also you'd want to update pre-existing list items, not just do a full data dump on each run.
However, in SharePoint, there really is no equivalent to GetDocumentByKey, so you cannot just grab and edit a document. Instead, you can write XML code to pass to a Web Service that will do the updates. I'll put together a code sample of that another day.
SPSite rootSite = new SPSite("http://localhost");
SPWeb site = rootSite.OpenWeb();
SPList l;
SPListItem li;
StringCollection sc;
//Create the SharePoint List and add fields.
Guid newGuid = site.Lists.Add("Notes Inventory", "List of Notes Applications", SPListTemplateType.GenericList);
l = site.Lists.GetList(newGuid, false);
l.Fields.Add("DB Path", SPFieldType.Text, true);
l.Fields.Add("Business Owner", SPFieldType.Text, false);
sc = new StringCollection();
sc.AddRange(new string[] { "SharePoint", "Exchange Mailbox", "InfoPath", "Domino Web UI", ".NET Development", "3rd Party Tool", "Decommission", "No Action" });
l.Fields.Add("Migration Plan", SPFieldType.Choice, false, true, sc);
sc = new StringCollection();
sc.AddRange(new string[] {"< 10 Hours", "10-50 Hours", "50-150 Hours", "150-300 Hours", " >300 Hours"});
l.Fields.Add("Scope", SPFieldType.Choice, false, true, sc);
l.Fields.Add("Priority", SPFieldType.Number, false);
//OK, we've now got a list -- let's fill it with Notes Data.
NotesSession s = new NotesSession();
s.Initialize("YourNotesPassword");
NotesDbDirectory dbdir =s.GetDbDirectory("YourServerName");
NotesDatabase db = dbdir.GetFirstDatabase(DB_TYPES.NOTES_DATABASE);
while (db != null) {
try {
db.Open();
li = l.Items.Add();
//create record
li["Title"] = db.Title;
li["DB Path"] = db.FilePath;
li.Update();
} catch (Exception ex) {
Console.WriteLine(ex.StackTrace);
}
db = dbdir.GetNextDatabase();
}
Yesterday I posted some code, and some reasons why it wasn't quite useful yet. So this morning, I fixed my main two complaints by adding check to see if the list already exists before creating it, and adding a check for an existing record before creating it. This could now be scheduled nightly, and will maintain your SharePoint version of a Notes catalog for you.
You would, of course, need to create some views to make the resulting list user-friendly. Maybe I'll tackle that another day, too. But that can be done easily through the browser interface, so it isn't a high priority on my list.
I included the full program code, as I realized some of the objects wouldn't be resolved correctly in the IDE unless you saw my 'using' statements.
One question, though - I did not find any method in the SharePoint API to check for the existence of a list by using that list's name. I used a try/catch block instead. Do any SharePoint folks know if that is appropriate, or is there a better method?
On with the updated code: (Disclaimer - untested code, but it should do the trick.)
using System;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Text;
using Microsoft.SharePoint;
using Domino;
namespace NotesInventory {
class Program {
static void Main(string[] args) {
SPSite rootSite = new SPSite("http://localhost");
SPWeb site = rootSite.OpenWeb();
SPList l;
SPListItem li;
StringCollection sc;
StringBuilder q;
SPQuery sq = new SPQuery();
SPListItemCollection items;
try //to get the list {
l = site.Lists["Notes Inventory"];
} catch (Exception ex) { // it isn't there -- make it
Guid newGuid = site.Lists.Add("Notes Inventory", "List of Notes Applications", SPListTemplateType.GenericList);
l = site.Lists.GetList(newGuid, false);
l.Fields.Add("DB Path", SPFieldType.Text, true);
l.Fields.Add("Business Owner", SPFieldType.Text, false);
sc = new StringCollection();
sc.AddRange(new string[] { "SharePoint", "Exchange Mailbox", "InfoPath", "Domino Web UI", ".NET Development", "3rd Party Tool", "Decommission", "No Action" });
l.Fields.Add("Migration Plan", SPFieldType.Choice, false, true, sc);
sc = new StringCollection();
sc.AddRange(new string[] { "< 10 Hours", "10-50 Hours", "50-150 Hours", "150-300 Hours", " >300 Hours" });
l.Fields.Add("Scope", SPFieldType.Choice, false, true, sc);
l.Fields.Add("Priority", SPFieldType.Number, false);
}
//OK, we've now got a list -- let's fill it with Notes Data.
NotesSession s = new NotesSession();
s.Initialize("password");
NotesDbDirectory dbdir =s.GetDbDirectory("server");
NotesDatabase db = dbdir.GetFirstDatabase(DB_TYPES.NOTES_DATABASE);
while (db != null) {
try {
db.Open();
//check to see if record exists
q = new StringBuilder();
q.Append("<Where>");
q.Append("<Eq><FieldRef Name=‘DB Path’ />");
q.Append("<Value Type=‘String’>");
q.Append(db.FilePath);
q.Append("</Value></Eq>");
q.Append("</Where>");
sq.Query = q.ToString();
items = l.GetItems(sq);
if (items.Count == 0) { //no record found -- create it
li = l.Items.Add();
li["Title"] = db.Title;
li["DB Path"] = db.FilePath;
li.Update();
}
} catch (Exception ex) {
Console.WriteLine(ex.StackTrace);
}
db = dbdir.GetNextDatabase();
}
}
}
}
A question has indirectly risen, both in online and offline conversations, of how we would define which applications should NOT be migrated to a new platform. Assuming that a small subset of Notes functionality will remain in the organization, what features or functions would lead you to not even attempt a migration? My answer would be the following:
1) Any application that truly needs offline access, and therefore replication. In my particular organization, surprisingly, nothing fits this category.
2) Any app that requires field-level encryption. The other granular levels of Notes security can be emulated with some skilled development efforts, although I acknowledge they are more difficult in Microsoft technologies. But I know of no way to set encryption on individual fields other than through the use of the Notes client.
3) Any app that is business critical, directly generates the revenue for the company, and also well-loved by the business users. An app that works, that makes the money, and that isn't a source of pain for people.... it should just be left alone to do its job.
Once again, comments and emails have given me some insight into questions and false assumptions that some Notes people may have. Namely, they assume that the recommendations given by Microsoft partners will apply to a Notes migration. And while certainly much of their technical advice is valid, the underlying viewpoint of a Microsoft partner is going to miss an extremely important piece of information about a Notes shop: The current collaboration in a Notes shop is NOT based around Microsoft Office
Notes people do not email spreadsheets back and forth. They do not use Access databases, or share documents via email. They already have a collaboration infrastructure in place, and are not starting from scratch. Sharepoint was designed to give organizations their first stab at collaboration. Its primary strength is to take Office documents, and share them on the web.
While I am sure this is a great help to many organizations, as a Notes professional, it just doesn't impress me that much.. SharePoint is an immature product. It has flaws and weaknesses, and we have already addressed some of them on this blog. But those weaknesses get exaggerated when we go out deliberately looking for flaws, and this type of exaggeration is exactly what I want to avoid here.
Let's call out the weaknesses, but lets not spin SharePoint's faults into something worse than it really is. With that in mind, lets talk about MOSS. MOSS is the "Microsoft Office SharePoint Server." It adds functionality to SharePoint. But not as much as people think. the plain old freebie SharePoint install is what gives you most of the programmability of SharePoint. And because a Notes shop is already used to doing a certain level of development for their apps, that is the level of complexity that most Notes shops really need. So when you see lists of the technologies from Microsoft that you must install and learn to run a SharePoint environment, take them with a grain of salt. Maybe in the future, your organization will reach that level of complexity. But to start a migration, you really just need a few things, most of which you probably already have:
What I am finding is that because SharePoint has limitations and weaknesses on its programmability, most Notes professionals do not accept its built-in functionality. So they step down into C#, and write custom .NET code. And while people will tell you that this is much harder than using the rest of the Microsoft platform, it greatly simplifies your infrastructure needs. (Not to mention costs.) Again - as your Microsoft platform grows, you probably will add on more tools and end up with a more complex infrastructure. But the assumption that it takes a ton of effort to get going on SharePoint just isn't correct.
One of the reasons that an organization would want MOSS instead of just the basic SharePoint services is for the Business Data Catalog (BDC). In short, this allows you to use your existing enterprise data as a data source within Sharepoint. No need to synchronize your list of customers or vendors between systems - just use your existing system. I can see where the BDC would become a critical component of your apps. But that is also exactly my concern - I can foresee it becoming a maintenance nightmare, as changes to your other systems can now impact your SharePoint environment. So I'm curious - has anyone out there actually been using the BDC long enough to develop any best practices on configuration & maintenance?
At the risk of being mocked for my lacking C# skillz, I thought I'd post another code snippet. (And at the risk of losing traffic, because it seems that whenever I post code, Google stops sending me traffic for a couple days.)
One thing we have encountered as part of our migration is the need to make a change to an entire environment. (Modifying ACLs, etc.) In Notes, AdminP can do some of this for you, but when the change needs some calculated logic, you need to script it. Notes has the NotesDbDirectory object which handles this nicely for you. But it got me thinking - what if you wanted to traverse your entire SharePoint environment? What would that code look like? So I sat down this morning, and spit out the code below. It is just a recursive call to each site's subsites, which gives you an opportunity to do something at each site:
static void Main(string[] args) {
SPSite rootSite = new SPSite(”http://yourserver”);
SPWeb site;
SPWebCollection children;
site = rootSite.OpenWeb();
DoYourStuff(site);
//Traverse SP hierarchy children = site.GetSubwebsForCurrentUser();
if (children.Count > 0) {
ProcessChildren(children);
}
}
static void ProcessChildren(SPWebCollection sites) {
foreach (SPWeb site in sites) {
DoYourStuff(site);
SPWebCollection children;
children = site.GetSubwebsForCurrentUser();
if (children.Count > 0) {
ProcessChildren(children);
}
}
}
Does anyone else despise Microsoft's documentation? I'm used to the help in Notes - it tells you everything you need to know about a given property or method, as well as a sample of the simplest script you would need to use that object. Microsoft tells you... syntax. And lists methods. No samples... or sometimes HUGE monolithic code samples that they use to illustrate eleventeen methods in one sample, and expect you to spend hours sorting through it to find which part applies to the answer you actually seek.
Well, in this case, the question applies to the SharePoint API -- specifically, SPField.TypeAsString. Very simple method, very easy to understand and use. But nowhere can I find a list of the possible return values. I found some sample code that gives me some of the values. And I could go create a list with a field of every type, and see what comes back... but shouldn't I be able to find this via documentation somewhere?
Update: I went ahead and wrote some code to see what would come back. I came up with the following list:
I normally refrain from talking directly about work, but this app deserves special mention because it is a rare one - it practically screams, "Migrate Me!". It gathers data from the end users, then lets them select their own approvers, and when approved, uses a Word Template to copy their data into a printable Word Document. The way it is done in Notes is very fragile. It uses OLE Links to the Word Templates. It creates multiple NotesDocument objects with the updated documents, and again uses an OLE Link. It then activates that link to display it to the user. Of course, if your files ever move, your app is broken. I can think of better ways to achieve the same result in Notes... But why even bother? If your requirements are for Word documents with a specific format, SharePoint becomes a clear and simple answer.
I have a few ideas I want to try in the near future. I don't have much spare time, though so I wanted to throw them out here to see if anyone has a reaction before diving in...
1) SharePoint API via LotusScript - I'm sure we've all seen and/or written code that calls functions from DLLs on our system. Well, SharePoint's API is housed in a DLL. why not call it directly? The only catch is that it is an OO platform, not procedural, so declaring your functions only gets you half-way. You would need the objects as well. I've never looked into the syntax, or the possibility, of declaring an object from a DLL in LotusScript. I will need to research it. If it can be done fairly easily, and you can in effect extend the LotusScript Object model with the SharePoint object, that really changes things. If not, Variants may still let you use a portion of the model, but it would be much more tedious.
2) Lotus Notes as an offline platform for SharePoint - I've designed, but not coded, a program that will take a sharepoint list, convert it to a Notes database, and keep the two in sync, allowing for a dual interface. It was mostly just an exercise to understand the two data models, but once I did it, I realized that it could serve a highly functional purpose - It could allow people to use Notes as an offline storage mechanism for their SharePoint data. How would that change the marketing of these products if Notes could consume any SharePoint list, and make it available offline just as if it were a Notes database? Like I said - neither of these ideas has much code behind them at the moment, but I think they could lead to some very powerful integration between the two systems. I have very little time in my life for side projects, but I'll keep people posted as I play with these and see what I can come up with.
(After some experimentation, I've concluded that yes, you could very well write some script libraries to extend LotusScript for your SharePoint migration. It wouldn't work as directly as I had hoped, but there is still quite a bit of value you could put together. My current projects are simple enough that it is not needed, but I will keep it in mind and hopefully build something over time.)
I am finding that the important part of building a SharePoint environment is not really technical, but that shouldn't be a surprise to anyone who knows me. Over more than a decade of Notes development, I've learned many aspects of collaborative development that are harder to define -- for example, having an understanding of the ways people really will use a system. Knowing which technical features they will gravitate to, and which ones sound cool to the developers, but will get ignored. Knowing which kinds of people want to work from a list in a browser, and which ones really do want everything in their email. And combining all of this with a basic understanding of UI design, and HCI.
I'm finding that many of the experienced MS developers have not worked directly with the end users enough to gain this understanding. There are some who have, to be sure.... but for the most part, MS developers seem to be used to working from a spec, and spitting out code to match. Some of them are very good at this, but the resulting application is only as good as the guy who wrote that spec. I've seen many apps that are beautifully, brilliantly coded, but that just aren't very useful.
This isn't specific to MS -- I'd say 99% of today's internet startups also fit that brilliant, but useless category. But the whole reason collaboration software came to be is to simplify and streamline the communications within an organization. How can an application developer be successful at this if they do not first understand the way that people prefer to communicate?
If you recall, back when I first started this blog, I wrote out a brief mapping of SharePoint features to Domino features. I now think that should be thrown away. The more I get into SharePoint, the more I realize a very fundamental difference between the two systems.
They both offer very granular control of your environment... but Domino starts at a high level with features that let you get more detail as you need it. SharePoint forces all the detail upon you, but roll it into an interface that lets you manage it.
For example, Security -- with Domino, you start with a basic ACL on the database. Everything works from that ACL. You can then add form level security, document level security, and field level security as your develop your app, but none of those are required. SharePoint on the other hand, really does store security information for each component. When you create something new, it inherits the security from its parent. You can (and often do) just leave it at that. But each list always has its own security properties.
It makes SharePoint slower to learn, in my opinion. A New Domino developer can learn how to build a form, a view, work the ACL, and be somewhat productive very quickly. They can always go back and learn all the details later. A new SharePoint Dev can quickly build a basic list with some simple columns, but to start working with security and workflow... that takes more effort.
Both systems have a great detail of potential. And a slower learning curve with SharePoint isn't really a problem - we just need to sit down and suck it up. But I do question long-term maintenance efforts on SharePoint. MOSS 2007 hasn't been out very long. It is too immature of a product to have established any real-world best practices yet. I'm surprised at how many corporations seem to be diving in to bleeding-edge technology, instead of giving the world a year or two to shake out the kinks. As a Domino folk, I still see SharePoint as the less mature product. So the management decision to put this much effort into a new technology that costs more, requires more maintenance, and has a chance of matching, but no chance of surpassing Domino still confuses me.
Nevertheless, I don't guide my career by a technology, but by business needs. If corporations make this choice, I intend to build my skills and give them the best system I can, even if I would have made a different choice.
It is too bad that I have a policy against writing anything directly about my projects at work. Because it stops me from getting into some details that would likely interest this audience. I can say this much -- I have been reading for a long time that running a Microsoft platform will cost more than a Notes/Domino platform.
After the past week, I now understand why that is. Readers in the past have questioned how long it took to get a SharePoint environment up and running. That is still the quick and easy part. The difficulty comes not in the setup... but in the maintenance. Maybe I just haven't figured out the best practices yet, but it seems like SharePoint is designed to require ongoing maintenance.
The theory is that the end users can manage their own sites, and IT just need to keep the environment running and do special projects. But wait -- this sounds exactly like the Notes world back in its early days. At that time, IT shops thought Notes could just be thrown to the users, and they can create their own DBs without much help from IT. And some shops did do that. But most of the places I worked did not... instead, the IT shop handled every single request for a new Notes app, they handled every maintenance issue, and eventually the Notes environments were locked down fairly tightly, with end users having freedom only within specific areas of applications pertinent to their business.
I would be surprised if SharePoint did not go that same route. Business users will probably really like being able to throw out a new document library, discussions, etc. But I don't think they will like creating lists. I doubt they will want to bother with the tedium of creating columns and views, and I doubt they will enjoy learning about all the different data types and function you can do in a list. If I had to predict the future, I'd predict that SharePoint environments will degenerate into a mishmash of document libraries, with a high-level structure generated by IT, and utter chaos once you drill down to the areas controlled outside of IT.
In a couple years, the major SharePoint projects will not be setup projects, but cleanup projects. We will spend our time reorganizing content, locking down access, and treating SharePoint like one large application (which it is.) The idea of collaboration gets confused with the idea of complete freedom within an application. Collaboration works best when there are enough controls in place that everyone works in the same way. SharePoint does have the power to create a good collaboration environment. But it needs IT oversight. Repeat after me - SharePoint is not a Sandbox.
I have not been posting very often in the past few weeks.... mostly because I've been swamped with working solely within SharePoint, learning some more of its details, and how it was configured in our environment. But I've been working 100% within the browser-based areas of SharePoint, so I haven't written any code to share or devised anything profound.
I do have a stronger sense of how to "think SharePoint". If you recall, a while back I wrote out a list of analogous Notes and SharePoint components to help me wrap my head around SharePoint, but I'm now getting past that. I'm beginning to think of SharePoint as if it were Legos. No single feature within SharePoint is very complex. You need to build them on top of each other to reach your goals. You have some moving parts that you use to connect your pieces and make them move in some limited ways. The more creative you are, the more you can make it do.
To work with the out-of-the-box features, you almost have to stop thinking like a coder -- it isn't code.
Its security is also more open-ended. While Notes/Domino has many layers of security, they all come down to the same IDs and groups. Not so in SharePoint. While you do often use your Windows Login as your primary identification and authentication, it gets blurry after that. You can use SharePoint groups, you can use AD Groups, you can use LDAP queries, you can write your own membership provider. In some places, you can mix and match, while in others you cannot.
I've always maintained that a Notes/Domino system is exactly as good as the people who design it. There are world-class systems out there and there are catastrophes. I think SharePoint is going to exaggerate this trend even more. You will find elegant, wonderful implementations, and you will find utter disasters, and the key will be the people behind it.
The problem is the level of expertise in the industry - Domino has been around a long time, each new version compatible with its predecessor. This has built a core group of developers who have well over a decade (or two) of experience, and can really put together a nice architecture. SharePoint 2007 is brand new. There is no expert with 10 years experience. The most senior folks out there are still working out the kinks of the latest version. And as each new version is its own beast, and not built upon the previous version, I fear that this will always be the case.
After a couple days of fighting SharePoint, and spending hours getting small details into just the right place, an image came into my mind. Imagine Notes/Domino as a trainyard - while it has a lot of power, and definitely needs some technical knowledge, once you are set up properly, you just need to know which switches to throw to get the results you want. Your 'train' smoothly goes in the direction you desire. Sharepoint is quite similar. Except that instead of a trainyard, your train is sitting in a big open parking lot, and you have lots of monkeys throwing crowbars under its wheels, and you need to see what happens and keep giving the monkeys new directions until you get the result you want. SharePoint is fun. Really.
Well, the previous post was not so unbiased, I will admit. But I'm feeling better about SharePoint today. I had a number of small problems to fix, and was able to fairly quickly and easily get into the site, make the udpates, get everything tested, and declare the problems resolved. We're starting to realize that going into SharePoint with a "Notes vs. SharePoint" attitude is dangerous. While they try to be competitive products, their architectures and techniques are so vastly different that comparisons inevitably leave us frustrated, wishing we could stay with Notes.
But that just isn't reality in our organization. So we are trying instead to discipline ourselves, taking SharePoint for what it is, finding its strengths, and enjoying the experience of learning a new technology. At the same time, as we look forward to next year, we are seriously starting to consider the details of application migration. Even though I don't have a huge audience here, I have received a few emails from other people involved in application migrations, and there seems to be a level of interest in sharing our experiences.
To that end, I'm wondering if I should expand this site into something more than just a blog. Perhaps add some places for us all to document our techniques that have worked well, as well as the lessons we are learning. I'd naturally expand the authorship of the site to let multiple people contribute to the content. Would there be interest in an expanded site along those lines?
A few months back, I had stated that setting up a basic SharePoint environment really isn't all that painful. And that is still true. However -- what I have found is that only basic dev or proof-of-concept environments actually stay that basic. We have about 2 dozen servers in our environment, for example. And we have spent the VAST majority of our time on administration issues, not on development efforts. In theory, everything will be wonderful once we get everything stabilized. But getting there is taking more effort than anticipated. Every time we work on one issue, we find three more. The theory sounds much better than the practice.
I think that SP will still come through for us. But the process to go from nothing to a full-blown stable multi-server SharePoint install that actually benefits a business organization... Whew. It is a doozy. There was always a half-true joke in the consulting world that to accurately scope out a project, you take the estimate from the technical team and double it. For SharePoint, I'd double it again. If you are hiring consultants to do it, double it once more for good measure.
I had mentioned quite some time ago that one feature SharePoint just didn't have was hide-when formulas. Let me state here and now that I was mistaken. Just this morning, I figured out how to do hide-whens in SharePoint. I don't have the full range of options provided by the @Functions in Notes, but I can hide form UI based on data within that form. I will try to expand on this later (I've already got 3 topics lined up to post about... just need to find the time), but in brief:
That's it. Now that you are working in XSL, you can use xsl:if to write logic to choose exactly what UI components should / should not display. This technique won't give you the full capabilities of Domino's hide-whens, as it limits your logic to the list data... but that is better than nothing, and still fairly powerful. I intend to do more research to see if I can pull in more environment/session/user data to expand the capabilities of this concept. And I suppose this also means I need to become much stronger with XSLT. I've suspected for years that XSLT is a skill I need to pick up. This forces the issue.
I wanted to go over a bit more detail about 'hide-whens' in SharePoint:
As I stated in the previous post, you can use a Custom List Web Part to pull your list item UI into XSL, giving you control of the UI, which in turn lets you hide elements using xsl:if. To begin, we need to add the Custom List Part:
At this point, you should see the specific fields for your list in your designer client. If you look at Design view, it looks like the default SharePoint form. Code view will show the XSL that creates the form. Now we can edit that XSL and modify our UI. By default, SharePoint will display each field in a table row containing the field name in the left hand column, and the field value in the right hand column. You probably will see many iterations of code that look like this:
<tr> <td width="190px" valign="top" class="ms-formlabel"> <H3 class="ms-standardheader"> <nobr>Field Name</nobr> </H3> </td> <td width="400px" valign="top" class="ms-formbody"> <xsl:value-of select="@Field_x0020_Name"/> </td> </tr>
Looks a lot like HTML, doesn't it? The XSLT tags surround all this HTML, so even through there is only one xsl tag in the above code, it all is programmable via XSLT if you choose to do so. (Also, note the field name - it converts your column name by adding a @ at the beginning, and converting spaces to '_x0020_'. That gets easier to type after you do it a few hundred times.
I thought about not using spaces in my column names, but while that would make the code cleaner, it would make the standard SharePoint UI less user-friendly, so I just deal with it.
Now, to hide something...
In general, to hide, you will use '<xsl:if test='YourCriteriaHere'>' So, to only show the value of "Assigned To" if "Is Approved" is set to 'Yes', you would write the following code:
<xsl:if test="@Is_x0020_Approved='Yes'"><xsl:value-of select="@Assigned_x0020_To"/></xsl:if>
Voila! Hide-whens in SharePoint. Naturally, you will want to do something more complex than that - if you choose to hide a field value, you probably also want to hide the full table row containing that field value. I also have used this to write custom text to the browser based on a combination of field values. Use your imagination. And Have Fun.
I want to throw a question out to everyone else who has worked in both Domino and SharePoint: In a number of our Domino apps, we will have a main document, for example a project description, with child documents, using different forms, for example, tasks, deliverables, meeting agendas, etc. Each of those forms may, in turn, have their own child documents. The final result is one parent document, with the ability to drill down in a view to see all the available child documents at multiple levels of a hierarchy. So the question -- How does one achieve this same type of UI in SharePoint? Is there any way to do it that isn't either spaghetti code or a maintenance nightmare?
I've been quite disappointed in SharePoint's mechanism for switching between views. A barely visible drop-down? Even if over time, SharePoint users might get used to that, our Notes Community won't accept it. Not when they are used to hierarchical navigators that display categorized views. So as an alternative, I went into one of my lists, and added a content editor web part to each page, and simply wrote up an HTML list navigator more along the line of what the users have come to expect. I'll see what the users think, but if they like it, I'll continue it. My way of doing it actually was pretty bad - it would be better to store the HTML somewhere either in SharePoint or on the file system, and call it via a different web part. But I'll figure out the best mechanism later - this first step is just seeing what kind of navigation works for the end users.
A little over a week ago I had 6 full posts I wanted to write, and was going to kick this blog back into high gear. So why did I lose my inspiration? A full week of literally babysitting the SharePoint environment, testing it every few hours, rebooting services and servers when they went down, just trying to keep it afloat. Tens of hours researching errors and problems, to find that we are not alone in our tribulations, but nobody else has answers either. Being told that other organizations had to rebuild their server farm from scratch to resolve these kinds of issues. Starting to do so ourselves just in time, as our initial farm dying a tortured death. Piecing things back together, getting 90% of the functions in place, but beating our heads against the last 10%. Hiring some of the top consultants in town only to have them shrug their shoulders at our problems. It is hard to be unbiased about a technology when you spend your Christmas just trying to keep it alive. I wasn't alone in this... one other member of my team has put in just as much effort and made sacrifices as well. Perhaps even more than I. So I have lost a lot of inspiration to write this blog. It still is my job to do this mirgration, though. So I will still write. But I have nothing unbiased to say right now.
Well, I finally got a day off from the disaster. Other team members are taking over from here, pretty much. At least over the weekend. One nasty problem is still under investigation, but we've resolved or at least identified the causes and fixes for the rest of the problems. A few lessons learned:
1) Have your server up to speed before setting up SharePoint. Be sure your patch levels are appropriate, and match on all servers.
2) The SmartPart is a cool little web part. BUT -- don't mix versions. Controls that work fine in older versions might not load properly in newer versions. Root cause unknown, but as it is an open-source thing, I'll let the authors worry about it. As a consumer, just pick a version and stick with it.
3) .NET Applications are more stable when left on the file system, with their own IIS App Pool. Pulling the aspx pages into SharePoint seemed like a good idea at the time. But not anymore.
4) Take notes when you set things up. We have a few web services and Search scopes that we need to recreate. If only we remembered exactly how they were configured... Imagine that. Documentation would have been useful? Who would have guessed?
Off-topic Post: I know that most of this audience is made up of Domino folks - so can anyone recommend a good Domino hosting company that is hosting R8 apps? I'm seeking a host for a startup project. "Good" means that they reply to inquiries in a timely manner, offer strong reliability, and are somewhat generous with their storage space and bandwidth. Cost isn't a major concern, quiality is. (I've sent inquiries to 2 companies already, but I haven't received replies, and in my mind, a hosting company that cannot respond to inquiries in a timely manner isn't likely to respond to support requests either.)
There has been a reason for not updating here in a long time. The burnout factor in trying to make SharePoint work is of course a factor. But more than that, we've finally started trying to architect our first major apps in SharePoint. That effort has shown just how short SharePoint falls. All the people who have been pushing SharePoint, saying that its "out of the box" functionality can build entire apps have realized that that is not true. The final conclusion is that while SharePoint can handle maybe 30% of any given apps functionality out of the box, the other 70% needs to be covered by BizTalk, .NET coding, and other Microsoft technologies. SharePoint is not living up to its marketing. The most ironic part of it all is that SharePoint shares exactly the same weaknesses as Notes. So if we did move entirely to SharePoint, we'd still have a system that is weak on reporting and on transactional processing. Our management team is re-evaluating SharePoint, and might reverse their decision to move there. Or might not. But the discussions have been re-opened. :)
After a few months of working with both Notes & SharePoint, I have updated my strategy to the following: Don't try to get rid of Domino - just get rid of the Notes Client. My proposal to do this is actually fairly simple:
Do those things, and the end users won't know or care what technology is driving any given app. IT still gets a cost benefit from not having to support the client, and management gets their wishes because "Everything is in SharePoint."
I'm looking for some answers from the community... As I mentioned previously, the status of Lotus Notes/Domino is again under discussion. To make an educated, unbiased business decision, we are looking for data on the following questions:
1) Development Effort - In General, how does development time in SharePoint compare to development time in Domino. Assuming skilled talent in both technologies, and non-trivial apps... does anyone have any real numbers and data comparing the level of effort to write applications on both platforms? Or even any anecdotal data?
2) Maintenance Effort - How does the level of effort compare to maintain SharePoint Applications vs. Notes Applications. (Again, in general). I'm not talking about the infrastructural support for the platform - just the application support. How many developers does it take to support a 500-app Notes platform vs. a 500-app SharePoint platform? If anyone has any real world experience, and in particular measurable data dressing these issues, please either comment here or contact me directly at <email address no longer valid>. Thanks!
After all of my inquiries and research into real measurable data comparing SharePoint Dev and Maint costs.... there is no data. Everyone I talked to is in the same boat we are - just getting going, spending more money on SharePoint at the moment, but with no real idea of what it will cost long-term. It is all guesswork. I've seen no evidence that anyone really is having a successful migration that saves them money. If anyone does have evidence, feel free to share it. But as I look hard at the two technologies, I see a few things... and this may show my bias, but it is what I see:
In general, SharePoint feels like a big old marketing scam to me. It doesn't do as much out of the box as Microsoft would have you believe, but it does give Microsoft and their partners a good chunk of money. A decision to go with SharePoint is a decision to tie yourself into their full product line.
Now, does that mean it is the wrong decision? Not necessarily. I don't know. It depends on your requirements, I suppose. It certainly makes me nervous, though. Why shake up the status quo for a new technology that nobody is skilled in, that costs more money to deploy, that is guesswork for long-term costs, and that ties you to a specific vendor? Even if it did perform as advertised, I just don't see justifiable answers to that question.
Welcome to all the new readers. I guess getting Brilled is somewhat similar to getting Slashdotted, if the 5000% increase in traffic is any indication. Thanks, Ed. :)
I'm mildly amused that when I start to border on ranting, the level of attention increases, but I won't worry about that today. I have other things on my mind. Now, this is all based off circumstantial evidence, not hard research, but given that caveat...
There seem to be two different ways organizations are approaching SharePoint, and they are diametrically opposed to each other:
Method 1: Install a server, see what it can do, use it for small apps, and expand its function and the scope of the environment based on demand.
Method 2: Assume that demand will be huge, and engineer a large, complex architecture to support that assumed long-term demand. (We did this.)
While I appreciate the strategic foresight of method 2, and the desire to not be constantly re-working your infrastructure, I have yet to talk to someone who went that route that actually has been successful. Up and running, certainly, but successful? Seems rare.
More often, the end users are happy with the platform, but there are internal teams like mine pulling their hair out.
Or maybe that does qualify as success, if you have the right perspective. In any case, folks going with Method 1 seem to be doing just fine. They are getting their feet wet with small stuff, learning the technology on the fly, as needed, and letting the growth of their environment march right alongside the growth of their skillset. Mistakes are made under both methods, but a small mistake in a large, complex environment, especially if identified after the platform has experienced immense growth into your organizations production systems, can be a real nightmare to fix.
I know we're suffering the pain of mistakes made in the planning phases that didn't show themselves until 6 months later. And of course, politically, it is unwise to say that the consultants for whom we paid a couple million dollars just plain screwed up. So I won't say that. And maybe I'm just looking at the greener grass on the other side of the fence. But seeing as our environment runs about 4 brochure-ware sites, front-ends a handful of custom .NET apps, and runs a few dozen each of lists, doc libraries, and discussions... I'm just not seeing why we couldn't have gone with method #1, saved millions of dollars, had a simpler infrastructure, taken less IT resources to support it all, and sure, maybe added a 2nd server if needed. So as a message to anyone else out there, when planning a new SharePoint deployment, KISS.
In the past week, I've received numerous emails, comments, and reactions to the general concept of migrating Notes to SharePoint. I wanted to post a brief synthesis of some of the main points, and add some commentary of my own:
1) Notes vs. SharePoint is not comparing apples to apples. Quickr is the better comparison tool. This is true. However, Quickr does not have the market penetration that Notes in general does. Much as I'd like to say it is a fair discussion for organizations in my situation, unless you already have Quickr going in your environment, it just isn't realistic. We're talking about determining the best platform for our current functions, not evaluating a new platform for future collaboration. Quickr just is not a consideration in our current environment, for political, if not technical reasons.
2) Web enabling an entire Notes environment is a difficult, expensive task. Maybe it is because I've been web-enabling Notes apps since the first IBM-internal beta of R5, but I just don't think it is as difficult as people are making it out to be. There is certainly a large learning curve, but I got over that curve almost 10 years ago. Am I the exception? From all the wonderful tips and techniques offered in the blog-o-sphere, I thought more Notes developers were comfortable in this arena. Is my perception inaccurate? That being said, I agree that a 100% web enablement would be a chore. We're more likely to pick the easy apps, and pick the apps with a large user base. If we can web enable enough apps to decrease the number of Notes Clients in the environment, we decrease support and maintenance costs. The more we decrease those costs, the more management will support keeping Notes/Domino as a technology platform.
3) Why would you migrate anyway? It all comes down to cost. I think everyone will agree that today, Notes is cheaper to run. We already have a talented staff, the environment is stable, we have processes in place, etc. In addition, truly talented SharePoint experts are rare and expensive. A lot of people have it on their resume, but they crash and burn in a real project situation. So with all that in mind, how can SharePoint save money? People I have spoken with believe that in 3-5 years, the costs will reverse. 3-5 years from now, SharePoint folks will be as common as C# folks are today. Microsoft will have released one or two more major releases, and will respond to the major weaknesses being identified in the platform. (With Ray Ozzie at its helm, no less...) On the other hand, Lotus folks are a dwindling population, at least in my area. I've talked to a number of organizations who will accept a greater cost today to get to the platform that they believe will become cheaper as Microsoft responds to the weaknesses of SharePoint, and the talent pool increases their skillsets.
After all the talks, my general strategy hasn't changed much from one of my previous posts, and it is very close to becoming the official strategy at my workplace:
Let me just add one or two caveats:
And I would like to close with an open-ended question -- given the following root causes of the desire of "management" to migrate to Notes...
1) Decreasing Lotus talent pool, and increasing cost of that talent.
2) Confidence that MS will resolve their SharePoint issues, and decrease long-term costs and pains in a SharePoint world. ....
What can IBM/Lotus do to turn around migrations like mine? (From a truly unbiased point of view, I shouldn't care. But with 15 years of Notes experience, but only 2 years of .NET, and 6 months of SharePoint experience, I just plain got more skillz on the Lotus side, and I'd like to use them.)
A number of bloggers have recently written their stories of how they got started with Notes. I didn't read them all, but the ones I did read share a common thread, and it is a thread that is applicable to the reasons people are migrating to SharePoint.
Every Notes person I know has been doing it for many, many, many years. I was in a room with about 50 Notes folks one morning, and the presenter asked every who had more than 10 years experience with Notes to raise their hands. Everyone had a hand raised. We are an "old" pool of talent. We are highly skilled, but expensive.
Young developers don't usually gravitate to Notes. I say "usually" because there certainly are younger, hipper, people who do Notes work. But they don't seem to stick with it, and they don't seem to become the die-hard evangelists that the online community presents. So its great that we have this well-developed talent. My last few years of work have been with wonderful teams, where any member of the team could have led the team, managed the projects, etc. It made for great working relationships, and we put out great work. I think those teams were worth every penny the customers paid.
But that is exactly the issue - without young, inexperienced, cheap talent... the hiring process for Notes folks tend to be unfriendly to budgets.
So what's my story? I started as an end user of Notes, when I got a job at IBM research doing desktop support in... must have been 1994. That job was just a temp contract, so I moved on to an analyst firm, where I did Notes admin for a while, sending their research to customers via Notes replication. (via modems to 300 servers. Ouch.) Again, that was a short term job, but I then moved to a hospital that was converting from green screen apps to Windows-based apps, and I was able to build up a small Notes environment to track the hardware assets, projects, and helpdesk work. I then moved back to IBM, helping them roll out Notes internally, and becoming a development team lead for one division. That led to senior developer positions in startup companies for many years, until I got back into the corporate world, being asked to help remove the platform. The one consistent pattern through all my career has been one of underlying changes in technology platforms. I've (almost) always been either build a Notes platform, or removing one. It is a technology that enables change on a very fundamental level.
For those keeping track, our formal strategy on what to do with Notes vs. SharePoint was on hold pending some hard data on the level of effort to develop and maintain applications. So we did what any group of people without data would do -- we made some ourselves. We selected an app that was basic enough to be realistic, but with enough quirky features to throw a wrench at us, and went ahead and made a prototype. Results were as follows:
And we concluded the following:
The bottom line is this -- from a purely technical point of view, migrating to SharePoint is a "neutral" activity. It neither offers great benefit, nor does it offer great harm. The decision is going to be made on a non-technical basis.
The cost of running SharePoint is likely higher today, but it will likely come in line over the next couple years as skill sets improve and Microsoft improves the product. Technical folks will be able to punch holes in SharePoint to their heart's content... there are enough gaps and issues to do so. However, SharePoint will meet a significant majority of requirements, and the amount of technical effort required to fill in the gaps will be an acceptable risk/cost to most organizations with valid strategic reasons to be moving to SharePoint. In particular, if a small/medium sized Notes environment is kept around for the ~10% of functions that SharePoint cannot handle.
Our overall recommendation in our organization is to do new development in SharePoint, web-enable as much Notes as we can, remove the Notes client from the organizations, and let the Notes system decrease over time through natural attrition, as the SharePoint environment matures. (Note that this is a change. When I started this blog, the strategy was "Get rid of Notes. Now.") This path also leaves us with options - if the platforms undergo major change for better or worse, we can always turn our strategy around.
For those of you who do not know him, Steve Walch has spent the last few years writing tools to integrate Notes/Domino with SharePoint, and now is working with Quest on migration tools. And now he is blogging: http://notes2sharepoint.org His blog is focused on his tools, and one of the differences between he and I is going to be our bias and focus. He states in a comment: "My #1 objective is to discuss the various issues that users of my product face on a day to day basis. My #2 objective is to discuss the broader set of migration issues that are not product-specific. You can expect this blog to be Quest-positive, Microsoft-positive, and indulge in plenty of "product pimping" along the way."
In my mind, this sounds great. It fills a gap in the online world today - Microsofties don't bother to talk about Notes, and Notes people tend to be in attack mode when SharePoint comes up. The reality is that migrations are going on, and will continue for the foreseeable future. Having a blogger around who will un-apologetically post content that tells us about tools and techniques to help these efforts is quite welcome, in my mind. I look forward to reading more from him as time goes on...
This isn't really going to be an in-depth technical post... just a brief description of a change of mindset that our Notes team needed to realize to work with SharePoint:
In Notes/Domino, we are used to a single agent holding all of the logic for a complex workflow. Between LotusScript, Java, Script Libraries, even calling out to system DLLs, an agent can become quite a sizable chunk of code. And we like it that way - it makes the agent list a readable, maintainable catalog of what code runs in each application. New developers can take a quick look at forms and agents and understand how an app works.
So in SharePoint, we were incorrectly expecting to see something similar - a single container that holds all the logic for a complex process. What we are finding is that expectation is making us fail at appreciating what SharePoint can really do. If we accept that a complex process may hold dozens of similar workflows to be called by separate lists, and come together into the full process, we are able to do much more work.
As the perfect example, from the day we sat down in a SharePoint training class, we were in shock that you could not have a workflow branch its logic based on form data. At least, not via the SharePoint designer. What never even occurred to us is that you branch your logic before the workflow starts. Each branch is its own workflow object, and you call the proper one. Our mindset needed to change from expecting a single object as the "parent" of all logic, to thinking of each object as its a single function of the greater whole, even though there is no place in the interface to point at as the container for the whole app.
We have gone through a fairly quick evolution on our opinion of SharePoint since I started this blog. I wanted to break it down, because it seems that people's attitude towards SharePoint can sometimes flag exactly how well they know the product.
Phase 1: "SharePoint can do anything. Let's migrate everything." These are the folks who have read the marketing, but never worked with the product. They just don't have a realistic sense of what makes it difficult.
Phase 2: "SharePoint sucks. It is a horribly immature product that cannot do much of anything, and any real complexity takes custom .NET coding." Noobs. People who haven't yet learned the subtleties of how to piece things together in SharePoint, and what kind of advanced customizations can be done with SharePoint Designer. (And yes, I was here when I started this blog.)
Phase 3: "Look at all of these cool WebParts I've written to help us get the most from SharePoint" Sadly, most consultants I see are here. They say they are SharePoint experts, but really, they are .NET experts who front-end their code through SharePoint. Their first reaction to anything complex is to fall back into custom coding. But they can do more with SharePoint than most corporate folks, so there are a lot of these folks out there building SharePoint environments.
Phase 4: "SharePoint can do a lot more than people think, if they know how to work it." This is the first level of expertise that I'd actually hire to work on a project. They understand how to do all the basics, and also can work the XML/XSLT to tweak the UI, they know how to create complex sets of workflows, lists, and libraries, then pull them all together into a slick interface. They know how and when to integrate InfoPath, and how to throw in C# code-behinds instead of giving us custom DLLs and webparts to deploy.
Phase 5: "SharePoint is just a tool. I have expertise with it and can make it do quite a bit, but let's evaluate it alongside other technology options and pick the best solution." Here we go. This is where we all want to end up -- knowing the strengths and weaknesses of multiple platforms, and matching solutions with business needs.
Of course, no one fits neatly into one of these categories. My team has elements of phases 3, 4, and 5 in their attitudes, but none of us have actually built so much experience that we'd claim expertise in the platform yet.
The reason this really matters to me is mostly for vendor relationships. If you are hiring a consultant, or even an in-house employee, I'd be very wary about anyone whose attitudes aren't at level 4 or 5. When looking at everything we've done in this first year, I'm concerned that we could have done things better, and have built some poor precedents for how we architect apps. We can certainly change the patterns we've fallen into, but that actually takes leadership skillz, and hey, we're tech folk. :)
When you add a new list item in SharePoint, it always seems to take you back to the list once you submit it. However, this is fairly easily controlled, in cases where you would like to send the user to a different page after they submit their form. (As in, when you would use a $$Return field in a Domino App.) The URL of a new list item, by default, is: http://server/pathtosubsite/Lists/List/Newform.aspx But take a look at the QueryString that is appended - you will see a "Source=" parameter. This is the URL from where the form was launched, and it is the URL that SharePoint will return the user to upon submit. Simply change that parameter to the URL of your choice, and your user will go there.
I've been quiet lately, I know. The reason for this is that I've been concentrating on refactoring Notes apps recently, preparing them for web enablement as they get integrated with SharePoint. But one issue we haven't yet resolved is Single sign-on, authenticating our Notes Apps with Active Directory.
I know the theory, and I've done it before - run IIS on your server, with the WebSphere plug-in, set some stuff on the Domino side, and everything is working. It usually takes a little trial and error to get it running the first time, but then all is well.
But here is my question - as part of configuring the WebSphere plugin, you give the hostname and port for your Domino server. Normally, Domino folks are putting the plugin on the same box that runs Domino, but... Has anyone instead put the plugin on their SharePoint server? Will it send the authentication cross-server to a different Domino box?
If so, would that then allow us to use the SharePoint domain name to reference our domino apps, and let the server actually process the correct routing of traffic? I'm guessing no one has tried this yet - but if it does work like that, it could be a very nice solution - we could have Domino apps thrown into our SharePoint site, and the end users would never know or care what the actual platform running behind the scenes is. This would let us maintain our investment in the Domino platform, while still reaping the benefits of a single SharePoint portal for the user interface.
As I'm sure most Notes folks are aware, SharePoint just isn't as mature of a platform as Domino. While it has quite a bit of potential, and you can bend it to your will, accomplishing almost whatever you want (if you toss in a generous helping of C#), it just doesn't offer the same day-to-day job satisfaction as doing solid development work in a platform on which you have years of expertise.
As one guy who commented a long time ago guessed, I'm just not sure that working in SP matches my long-term goals. I've always thought that I should complete my current migration effort, then switch to a new job, or a new role within my current organization. Some recent events at work (not related to SP or Domino) have made me wonder if I should speed up my efforts and just switch.
Purely from a job satisfaction perspective, SharePoint is drudgery. Learning new techniques amounts to a little bit of time reading up on how they are supposed to work, and a lot of time working around the quirks about how it REALLY works. I've been downright thrilled with my recent Domino work on the other hand -- I'm working with apps that have been around for many years, and have become heaps of mangled spaghetti code. Cleaning them up, re-designing to match the current business needs, and converting them to web interfaces... Truly enjoyable work.
It means I haven't had much to add to this blog recently, I know. There are plenty of people already covering the Notes/Domino world as-is. I'm sharing these feelings with all of you not because I think it will help you with your own migrations, nor as a 'fishing' post to see if anyone wants a consultant for a while, but just as a statement of how your average Notes guy might feel 6 months into a migration. Morale is definitely an issue to worry about on migrations, as frustration levels run high. I know if I were my own manager right now, I'd be worried a lot more about the people on the team than the status of the project.
A while back, I had asked if anyone was using the IIS process on their SharePoint servers as the front-end to their Domino apps, in order to achieve Single Sign-on with AD. The responses were positive, but I didn't have a chance to try it until today. And Yes... It Works. The instructions that are in the Admin help, and out on notes.net in the various forum threads get you 90% of the way there. A few other tidbits that I had to discover on my own:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?> <configuration> <system.web> <httpModules> <remove name="PublishingHttpModule" /> </httpModules> </system.web> </configuration>
Now I can happily render all my Domino apps in a SharePoint Web Part, and our end users don't have to know or care what back-end systems actually run the apps. Of course, I have a veritable heap of work to do to make the apps work on the web, and I need to write a script to handle moving Active Directory IDs into the person documents for everyone, but all the infrastructure pieces are now in place. All we have left is a lot of coding. A LOT of it.
UPDATE: I found a 64-bit version of the plugin, after much searching. All is now well with the world again, and everything is running. If anyone else does hit this, search IBM's web site for WebSphere fixpacks - the DLL that worked for me is found in FixPack 17 for WebSphere 6.1. Oddly, the broken DLL is in the /bin directory of that fixpack.. I needed to dig down into the other directories to find another copy of the DLL that actually worked.
-------------------
I take back my last post. Partly. SSO works great.... on systems running IIS in 32-bit mode. Like our development environment, where I did my initial testing. But... our test and prod environments are 64 bit, and the ISAPI filter fails to load. I learned a bit about IIS various operating modes, and based on what I know so far... I cannot leave SharePoint in 64 bit mode, while loading a 32 bit ISAPI filter into a virtual directory on the same web site. Which means... it doesn't work. Now, I could go back to the original plan and put the ISAPI on the Domino servers. But that is not an optimal solution, partly because of the logistics of the controls processes surrounding the various environments. It isn't impossible... just a very large bureaucratic headache that would take a couple months to get done. Gotta love the corporate world. It also is not preferred because we would ahve to switch the port on our Domino HTTP process. Which would break our existing links. Putting it on the SP servers lets us update our linked content as we are ready, giving us a very flexible transition on to the new architecture. I don't want to lose this benefit. But I find it difficult to believe that we are the first place to try running the WebSphere plugin under 64 bit IIS. Someone else must have hit this problem before. My searches to date haven't found anything, which means either I am missing something glaringly obvious that would solve the problem, or everyone is sticking with 32 bit. Or there is a 3rd option that I have not thought of. If anyone has any insights into this nice little twist of technological fate, please enlighten me via the comments.
This might be a bit of a stretch, but bear with me...
As programming languages have developed, their evolution has become more like natural language, more intuitive, and well... a lot less like assembly. I feel that systems such as SharePoint are really just a continuation of the abstraction of the code to a high enough level that people can create applications systems without even writing code at all. These systems are 100% configuration, 0% code.
But... that does not mean that the lessons learned via decades of programming practice across the world should be thrown out. Rather, if we can apply software engineering concepts to the design of our SharePoint systems, we likely will have better systems. Am I wrong?
The specific example that just came to light was the way content types were set up on our intranet. I inherited this SharePoint environment after some high fallutin' consultants took their checks and ran. But when I saw the way they had designed some specific aspects of this portal, I emailed them basically to ask what the rationale behind their design was. Amongst the actual answers was some advice/instructions:
Create all content types at the root level of the portal. Any new content types that inherit from the root content types must also be in the root of the portal. All changes must be done in the root of the portal.
Now, I get it - they want these to be available to every possible sub-site. But... doesn't something here sound quite a lot like, "Make sure all of your variables are declared globally."?
Admittedly, Content Types are not variables. They do not allocate memory just because they exist. Issues of variable scope don't apply. This is more of a readability/maintainability issue.
It just seems to me that any type of function that is specific to one area of a site could very well reside in only that area. Otherwise, as I mull over what the portal might look like in 10 years, I can easily envision a nightmare of content types in the root, with no intuitive explanation of where each is used, what it offers, or why it was created. Is that really what we want?
<petty rant> On another note, I'd like to tell all consultants out there -- Content Types are not category fields. If the only thing they do for you is allow you to categorize list items in the UI, don't create a content type for that. Just make a new column. Call it... I don't know... maybe... Category? Don't turn Content Types into your personal hammer just because it is the only feature of SharePoint that you understand. Now run along and spend those big checks we gave you for your brilliant insights into this technology. </petty rant>
The point of all this is really to try to express the need to design SharePoint like you would design any application. Not to just treat it as a collection point for hundreds of mini-apps, but to really think out the long-term design of the environment, and create each function acorrding to a broader vision.
As I've mentioned in "recent" posts, we've been setting up the WebSphere plugin on our SharePoint servers to enable authentication to Active Directory for our Domino apps. It has been in production for a few weeks now, and is working great. Not only have we received no complaints, it is actually resolving helpdesk tickets for us, as Domino internet passwords have always been a thorn in our side. The last change we did was to add a JavaScript redirect on to our server-wide Login form, so that anyone who does try to hit the Domino apps directly, instead of through our SharePoint URL, will be redirected to the proper URL. This is allowing us to ensure that SSO is used universally, without having to worry about whether or not we forgot to update some links on other web sites or in email notifications. Our next big integration point is to get our Active Directory data more accessible to Notes Applications. Currently we have nightly processes that pull the data from AD into a Notes DB, but there are too many moving parts in the process, and we are constantly fighting data issues. I've written some code that uses the XMLHTTP object to call an AD web service, so I can pull data directly from AD on demand. We'll probably also pull data down into the NAB on a regular basis, but being able to replace @dblookups with web service calls to the true data source (where performance allows) will be lovely.
Not much posting recently because nothing profound has been going on. We've been moving Notes apps over to SharePoint, doing easy stuff first. Simple forms and workflows that can turn into a sharepoint list, calendars, discussions, etc. Just a couple tidbits we've learned:
1) If a user needs to see the old data for a fixed amount of time, we decided migrating the data to SharePoint is wasted effort. Instead, just give them a web interface to the Notes data, either in a new window or an Iframe from the SharePoint UI. Customer are very pleased with this.
2) Icons in columns - We don't think much of this as developers... sure, we can put up a green checkbox or some other icon in a Notes column. It is such an easy feature, it threw us for a loop when SharePoint couldn't do something so simple in a column. But it can be done with Dataviews. So we've had to pick up some more Designer and XSLT skills to match the options available to us in Notes. Same thing when trying to calculate anything based on "multiple lines of text". Can't be done as a standard list option... needs to be done via XSLT in a dataview.
But the real reason I wanted to post was to share our new "Bad" answers to strange, inexplicable problems in SharePoint. We realized that, as many Notes Devs know, sometimes a problem just isn't easy to diagnose, and some folks have standardized answers that sound plausible, but which are totally false, and just serve to get people off our back while we figure out the real problem.
I've gotten over this bad habit as I've aged, and now freely admit, "I dunno... lemme go figure it out.", but I still hear plenty of Notes folks who says things like. "Hmm... there must be some database corruption. I'll go work on it.". Which often means the same thing as "I dunno."
So in jest, we came up with the two following statements in cases of unexplained SharePoint problems when you just want the boss off your back so you can go find a real answer:
1) "It sounds like an Office 2007 integration bug." - for end-user problems.
2) "That sounds like a timer job issue." - for server-related issues.
One of the challenges we've encountered as we migrate application off of Lotus Notes and over to SharePoint is what I call the "data infrastructure." There may be a better term for this, I dunno... but I am talking about those Notes databases that are shared data repositories for the enterprise, but not applications unto themselves.
For example, we have a database with a record for each employee, with their Job Title, Location, Phone, etc. We have another one for just IT employees that tracks who falls into specific support roles for each office location. And yet another that holds generic keyword lists... our office locations, department lists, etc, etc. Some of this may be stored in the NAB in some organizations, too... but that doesn't change the problem -- one way or another, you need to replace these data sources with new ones.
For some (most) data, just a new SharePoint list is sufficient. Or multiple lists. But the hassle comes in data conversion, and/or data maintenance, especially if there is yet a deeper root data source that was populating the Notes data. What I have done to make this process go forward is the following:
One of the key pieces of information that got me going down this path was the (not-so-profound) epiphany that AJAX calls are not limited to web browsers. They work great in the Notes Client, too. The XMLHTTP object is just an object in the OS like anything else: Set xmlhttp = CreateObject("Msxml2.XMLHTTP")
So I just declare the object, call my Web Service's URL, and parse the resulting XML for whatever purpose is appropriate. Once the data is available in LotusScript, any existing code can take over from there. I haven't thought of any efficient way to make this work in @functions... I suppose you could use @URLOpen to pull in the Web Service data, but I suspect by the time you got done parsing everything, it would have been easier to just re-write your code in LotusScript.
A year ago on this blog, I listed some Notes functions that did not have equivalents in SharePoint. Over this past year, as I learned more, I learned that I was just plain wrong. So let me take the time to correct my statements, and lay out the functions in SharePoint to replace the following Notes functions:
There ya go - just reinforces my point that even the most open minded Notes person will not "get" SharePoint development right off the bat. It take times to learn the new moving parts and figure out how to put them together...
There is one part of migrating to a new platform that I am truly enjoying -- the cleanup work on the Notes/Domino system.
I love taking an old, cumbersome app, and refactoring the code into a quick, simple design with significant performance and UI improvements. I can usually reduce the support needs at the same time, frequently reducing the headcount required to support the apps by well over 50%.
We end up doing this kind of refactoring on many apps before thinking about a migration because many of the apps have have devolved into such massive heaps of spaghetti code that they need a good refactoring just to understand exactly what the current functions are. Once the refactoring is done, then we talk to the business about whether all components are still needed, and look at whether or not a change in a business process may allow us to simplify the app even more.
What we find is that through this simplification process, the migration becomes only partially technical, but as often as not, a business transformation is the result of the technical work. A a chunk of the time, the resulting app is so easy to use and maintain that we just stick with it, and don't migrate it after all.
Which leads me into my next train of thought - if the refactoring is what I enjoy, and what I excel at, why do I spend so much of my time on the areas that I do not? So I am considering taking on side projects from other organizations. Seeking out other people, whether or not they are migrating, would would have an interest in hiring me to refactor their "ugly code". For now, I am thinking just 5-10 hours a week on the side, unless someone has a project so major that it makes it worth switching to an independent consultant on a full-time basis. So what do you all think? Is that something you would have any interested in contracting out?
Even though I know the "masses" reading this blog only number in the few dozen a day, I wanted to pose an admin question: Is there any way to verify whether or not a roaming user in Notes actually uses their roaming capabilities? We're down to the point that 75% of the databases on our production servers are roaming databases, and I know for a fact that the vast majority do not use roaming. But I don't know of any way to figure out exactly which users comprise the minority that need to keep those functions... Any ideas?
It may have been quiet on here, but I've been busy cleaning up our Notes environment. And we have hit what I feel is a nice milestone -- we have under 1000 Notes databases left on our productions servers, and that includes the system databases and others that I know we don't need to worry about migrating. Another 500 should be gone at the 1st of the year, and I'm hoping to enter 2009 with only about 250-300 databases to worry about. (Of course, some of those 250 are very large, complex, apps.)
I'm having fun this week. I'm digging into InfoPath for the first time, as we're now into some of the more complex forms that really are not readily handled by basic SharePoint lists. I've discovered a few tidbits that I wanted to document here.
Please note that I'm a flippin' newbie when it comes to InfoPath, so some of these notes might be obvious to others, and some of them might be bad ideas and show my inexperience.
InfoPath is an XML-based product. Underlying your forms is (normally) an XML schema. This actually translates very well to Notes data. Notes parent/child relationships between documents mesh very nicely with an XML scheme that includes child document data as a node in the XML. What this means is that I've been able to take multi-form Notes databases, and replace them with a single InfoPath form, published to a SharePoint library. However, this approach did have a few challenges:
1) User Interface Putting all that data on one form is a UI challenge. In the two forms I've worked on this week, I've used tabbed tables. One is your typical tabbed table, with tabs across the top. The other was a very long form (about 30 pages printed), so I created vertical tabs down the left side of the page that act as a table of contents for the form. There are articles out there on how to do tabbed tables in InfoPath. Go read them to get the idea of what controls you need to lay out, then come back here, because I did it a more efficient way. Instead of using your buttons to switch to a new view, think like a Notes Developer -- Use "Hide-Whens". In InfoPath, it is called "Conditional Formatting". Create data fields to store which sections should show, and have your buttons set those fields appropriately. Then just set your conditional formatting off those fields, and you have tabbed tables in a single view. This also has an added benefit that you can create multiple tabbed tables that all operate independently of each other. The view-based solutions online do not allow for this. "But wait -- isn't that mixing UI data into your XML? That is a horrible thing to do." Yes. Yes, it is. First off, relax. We're already on Microsoft products, so whatever you do is a hack, no matter how clean you make it. :P But seriously, I did make an attempt to address this issue. I created a new node in my XML to hold this form metadata. If I need cleaner XML to send this data to another system, I just need to drop that node. And in the meantime, my UI holds its state between form loads. XML purists may hate my solution, but XML purists probably aren't corporate IT grunts doing a job like this anyway.
2) Security We don't have authors fields. We don't have readers fields. We need to make SharePoint security do everything for us, or use pseudo-security via the UI. For SharePoint security, we're kinda out of luck for Readers/Authors functions, but it can handle most other security needs. For UI security, it isn't truly secure, but depending on the application needs, it might be secure enough. InfoPath and SharePoint are not like Notes, where a savvy user can edit the document data by making up their own forms and agents, etc. You cannot right-click and see data fields, bypassing the UI. Even for most savvy users, they have to use your UI. So if you, for example, disable all controls in your InfoPath form unless the current user is listed in a specific field, that will actually work more reliably than a similar technique in Notes. There are still ways around it, but it should be enough of a deterrence to be effective for basic business apps, especially if you have revision history turned on in the SharePoint Library and can throw out any inappropriate edits to your documents.
3) Workflow InfoPath coding cannot send email notifications. But it does very well at creating buttons that perform different actions. SharePoint workflows are more complex, if they need to perform logic on your InfoPath data, but they can create workflow tasks and emails very nicely. I haven't yet got all the answers as to what the best practices will be when making choices between InfoPath rules and SharePoint workflows. For the form I finished today, I did most of the work in InfoPath, then created a SharePoint workflow to handle notifications for the approval cycle. To make your InfoPath fields available to SharePoint, you must select them when publishing your form. It is hard to miss this step, as the publishing wizard asks you very directly which fields you want to expose to SharePoint. Be sure to expose anything that your workflow will want to read. If your workflow also needs to set an InfoPath field, there is a checkbox that reads "Allow users to edit this data in a datasheet", or something to that effect. Select this checkbox. If it is not selected, you cannot set the field with a SharePoint workflow. If you have multiple fields in your XML data source with the same name, but under different nodes, SharePoint pulls them all in with just the name, and loses your hierarchy. I didn't delve deeply into how much this would screw things up, I just got a sense of ugliness and decided that I needed a different way to handle things. So I made copies of the data I needed in my meta-node that I used for UI data, and I exposed these copies instead. Is this a bad idea? I don't know. There may be a cleaner way around this particular problem, but I have not yet learned it.
--------------------------
In general InfoPath is a nice tool. It will feel more like Notes Form development to us Lotus folk, and provide many of the features we are used to. Much of our design skill and UI experience will translate directly across, and I wouldn't be surprised if Notes people start becoming very innovative in InfoPath, as we think differently than .NET folks, so we will approach form development with new ideas. I'm looking forward to getting past this phase of learning, and seeing what I can really do with the tools over the next few months.
So in Notes, popping in a Names field with a default value of @UserName is a very simple, easy action. In InfoPath? Not so much. I first had to seek out how to even get a name selection control running in an InfoPath form, and then I discovered that the XML data needed for that control made a simple userName() call ineffective. A plain-text field can easily pull the current user, but not a selection control.
So I spent some google time compiling solutions, and found how to make the control pull the current username. Cool. But the answers I found pulled the name on all form loads, not just the initial load. So I had to get the rule to run only on the first load. Doing field comparisons to only run if my values were blank didn't work. XDocument.IsNew was not recognized as a valid formula. So I ended up doing another hack with inappropriate XML data to flag what status the document was in. I'd like a cleaner answer to running a Form rule on initial load only -- does anyone have one? In the meantime, here is the ugly detail that I emailed to my co-workers in case they need to do the same time:
If you want to add a field to an InfoPath form with a name selection from AD, follow instructions here:
10) Set the Rule to only run on the initial load of the form
Whew! Now you are done and it should work..
---------------------
Just as a comparison, In Notes:
It is coming up on the end of my first full year doing this migration. I started the effort last fall, but all my time in 2007 was dedicated to learning SharePoint. But it is now time for the annual corporate performance review. Everyone in the organization now tries to pimp themselves, pontificating on all the wonderful accomplishments from 2008, trying to get that extra 50 dollars on their bonus check that a good review will offer. And I'm finding this time to be a strong validation that the migration is going well. Because people are trying to jump in and find ways to include my work into their own accomplishments.
It doesn't bother me -- I already know the ethics of those in question, and they are doing exactly what I expect them to do. But, yes -- it was a good year. 12 months ago, I could barely do anything with SharePoint. I now am comfortable that I can make SharePoint do what I want it to do, and when adding InfoPath into the mix, I can make either the Lotus platform or the Microsoft platform perform well. I've done a number of projects on SharePoint, an equal number in Notes, and cleaned up both environments with the help of my peers. I've eliminated 80% of the Notes environment, with high hopes to do the other 20% in 2009. (The first 80% was the easy stuff.) Of course, if I do succeed in that, I'll need to find a new gig in 2010. But that is a long ways off...
So this has nothing to do with Domino nor SharePoint, but it is something nifty that I thought I'd share: I had a need to modify an Excel form sent to me by a vendor. The form was password protected, so I did some quick searches to find tools to remove the password. To be honest, I didn't trust the tools that I found. So i tried the following procedure. It works great: 1) Open your Excel file in OpenOffice 2) Re-save it as an Excel file. 3) Re-open in Excel and now you can remove protection without being prompted for a password. Thanks, OpenOffice!
I really wanted to be able to thank everyone who blogged at Lotusphere and helped keep us all in the loop. But I cannot... I do thank those of you who posted actual content. But I need to rebuke those of you who failed to recognize your audience. I am not going to poke fingers, but a number of folks "live blogged" various sessions, and the content from your blogs goes something like...
Seriously, folks. For today's readers, or when you come back in a couple years to read again, do you really think the valuable content came by listing what questions were asked? Without telling us the answers??? Back to your regularly scheduled Notes & SharePoint blog next week.
As we are now a few weeks into 2009, I've been thinking carefully about what direction I should take my career. The current economy doesn't leave me many choices, so this is mostly an academic exercise - the reality is that I will just keep my current job for the foreseeable future. But, were I to have my choice, I think I would choose not to pursue any more SharePoint skills or jobs. I've done enough of it now that I know its good side, and its bad side. And I'm just not enjoying it. My time is either spent doing a lot more configuration than coding, or I am fighting the platform with code to try to bend it to my will.
I like using technology to solve problems, not fighting technology to create basic business process applications. SharePoint does work. .NET does work. BizTalk works, InfoPath works, the whole Microsoft platform works. But it just isn't a satisfying platform to work under. When I was 100% dedicated to Lotus Notes, I could have some passion about the platform. But under Microsoft... my job is just a job. This weekend, I just downloaded Django and did a few basic projects. It was quick, it was easy, and it was fun. If Notes jobs aren't around, and SharePoint is boring... I may have to spend my time in the open source world. When the economy gives me the luxury of choice, that is...
I'm always very resistant to buying tools for development work, and that is part of the reason that I have not pursued any tools for this migration. The other reason being that, while we are migrating away from Notes, it is intended to be a slow attrition, not an active push to get rid of the platform, so we don't exactly have a budget to make it happen. We're just doing what we can with our own internal resources.
In any case, I dislike most tools for two major reasons:
1) They don't really do anything that a good developer could not have done on their own -- I don't like buying a tool when I could write it myself in under a couple hundred hours. The migration tools that I've seen tend to fall in this category. They are slick, and work for many people... but if I can write an ugly simple script that does the same thing exactly as I need it to... why pay money for the fancy version, when I don't need its functions? It actually make me somewhat angry when another team member recommends we spend thousands of dollars on something I could do myself. My "Developer" personality comes out, and I feel that I am not valued by the team. Tools like this are not only a waste of money, but insulting as well.
2) They tie you to the vendor and/or product: ...SharePoint itself falls in this category, but I've already expressed that I do SharePoint as my job, not my passion. One of the biggest roadblocks in getting off of Notes right now is to disconnect ourselves from some of the 3rd party tools that have been put in place in our environment. Some tools are made not to solve a one-time problem, but to fundamentally change how you create your applications, with the tools now embedded in your platform. So it becomes very difficult to stop using a vendor. I have worked in product-based companies, and the leadership was always very exited when we could write features that made the products "sticky" within the customer's environment. While that is great for the vendor, it is not good for the customer.
These two points alone make me shy away from most toolkits I've seen. But not all -- there are great tools in this world. When I find them, I love them. Or sometimes, a tool will still break my first rule from above, but only cost a couple hundred dollars, so it is still worth the money. I have not had time to deeply investigate all the tools available in the SP/Lotus world. I suspect some tools do fit my criteria as being worthwhile, but I don't yet know which. If anyone does have any recommendations for tools that are worth their cost, please share in the comments...
I got bad news today at work... No, not a layoff. But the economy has shut down some projects, as I may have mentioned before. Specifically, it shut down projects there were intended to replace my largest, most complex Notes apps. These two apps are the ones that will make or break this migration, as they are used across the entire company, and too complex to migrate without significant custom development on a new system. They also are no fun to maintain.
I commented on Twitter that I should start a Notes Dev Hall of Shame, and almost all the code I am thinking of comes from these two apps. My mind reels at some of the crud I am finding. Oddly, they are not all bad. One app does have some really good code in it. But it has been shuffled from developer to developer over 12 years, and the last good developer looks to have been around 6 years ago. Admittedly, that guy was awesome. His code is good, readable, and his comments not only tell me what is going on, but make me laugh, too. The guy had a great sense of humor. Sadly, his code is sparse in the app at this point.
In any case, the migration suddenly looks less likely to be completed in less than 3-5 years now, even if I do everything but these two very quickly. My job is looking to turn into a stagnant Notes development job, working on an old version, maintaining old apps, with nothing new on the horizon... Well, nothing new other than SharePoint. I'm sure it will not come as a shock that the SharePoint work doesn't excite me. But when I was hired, I thought I was walking into a great team, with good code, and a company that was going to put some resources behind moving to a new platform, while gaining new skillsets to help expand my career options. Now that I know that, well, none of the above is true, it is becoming more and more difficult to stay positive about this job. I do recognize that many people, including some excellent folk in the Notes community, are seeking jobs, so I am thankful to have a stable job. But long-time readers of this blog will recognize that my tone just can't stay positive. I'm craving a real development job, not sitting in a cube farm doing maintenance work on a platform going nowhere. Especially when I know that both Lotus and Microsoft offer directions for the apps.... but we aren't choosing either path. We're sitting. And sitting. And taking tech support calls. And fixing some bugs. And sitting.
I'm thinking I may have to look for new work, but recognizing that the odds of actually finding any are slim at this point in time. But to combat this negative lethargy, I believe that I will start sharing some of the crud I'm finding very soon. I decided that I don't have the time to create a separate site for it, but I'll create a new category on here. This may change the overall theme of the blog a bit, but with the migration slowing down, there isn't going to be a whole lot to share anyway over the course of 2009...
I'll pull actual code samples for later installments of our Hall of Shame, but first I simply wanted to share a few choice items that have not only boggled our minds, but also caused endless amounts of helpdesk tickets.
1) The EmployeeID field in names.nsf At first glance, it almost looks reasonable. They added a field to the person document, labelled "Employee ID", called EmployeeID, and updated nightly from our HR system. All applications look to this field when they need to get user data, as it is the common data point between every platform in our enterprise. Or is it?? As we started to get all kinds of calls that applications were breaking, we found that about 20% of the applications read this field. The other 80% read 'EmplID'. Also in the person doc. NOT displayed in the UI. Nor was there an agent to set it. Our old admin team set this field manually when they created new users, using agents local to their system. Only when we switched to a new admin team did the problem crop up. The new admin team was already setting the visible field, and we ended up hacking together a little agent to keep both EmployeeID fields in sync.
2) The Action bar of Doom and Despair Let's ignore for one second that this action bar has 125 actions, and all of them are the same code, copied and pasted with minor modifications. Let's even ignore that any given user only will see 2-3 of these actions, based on their role within the system, so there may have been a simpler way to do these functions. Lets just focus on the hide-whens. The hide-whens that repeat the same DBLookup 5 times in each formula. Repeated across 125 actions. With some actions having 15-20 DBLookups, instead of the paltry 5. Do the math -- I counted 724 @DBLookups in this action bar just to determine which actions should display. The performance was so bad that people wouldn't even open the form. When I took this job, I was told that it takes 10 minutes to open that form, so they installed a Citrix system... because via Citrix, it is much faster. Really, is that the answer to an application coded so badly that it is unusable? To install Citrix to avoid the network bottleneck? I rewrote all the code into a single function, wrote an agent that does all the lookups in a batch process each night, and sets your personal access in a profile document. One lookup to the profile document to know which actions to display. One set of actions with dynamic labels and a function call. I was left with 10 actions, 1 function, 1 agent, and 1 lookup. And the form now opens in 3 seconds.
When people say "SharePoint", they usually mean any application accessed through a SharePoint portal. In truth, this may actually be SharePoint, but could also mean .NET or InfoPath applications. One of the challenges we've had is defining which types of applications really are a SharePoint app vs. InfoPath vs. .NET.
But before I go there... why does it matter? It matters in our organization because of the various roles certain individuals play. Power users and business analysts can do some of their own work... if it is truly pure SharePoint. We're finding that Domino developers pick up InfoPath so well, they can do more with it than many Microsoft experts. (Which is a blog topic in and of itself, but in short, we've spent years working with form-based platforms... we know what to do with, for example, Conditional Formatting (Hide-Whens) to simplify forms instead of making new forms and views for minor UI differences.) And... .NET -- while our team can certainly write the code, it brings on whole new levels of deployment and maintenance issues. So we try to avoid it.
Which brings us to the "Road Map". Which is not very profound... it is a very simple thought exercise for any application function that we are asked to provide:
1) Can SharePoint do it? Then make it so.
2) Well, then, can InfoPath do it? Then make it so.
3) Well, then, we'll have to write .NET code.
After some discussions, we also added a question 0: Does this even need server-side code?
If JavaScript in the browser can do it, don't get too clever with SharePoint... just write the script. We had to add this question because developers who have never really been web-focused are forgetting that SharePoint ultimately renders in a web browser. Throwing some script in a master page or even in a content editor web part is sometimes a very quick, easy solution to some problems. Our most recent example of this was a 'Contact Us' link. It was supposed to be available on all pages, provide an infopath form, then return the user to their original page on submit. They first wrote .NET code to do this. I talked them into just using a SharePoint list. Then they were doing some crazy SharePoint CAML stuff to make it happen. I talked them into just using JavaScript to append '&Source=originalpageURL' to the HTML link. I use this as an example to illustrate the eternal point of KISS -- if 1 line of JavaScript code does the trick, don't go writing .NET features. Find the simplest solution.
The main topic for today is user-configurable data. It is true that hard-coding data is bad. The values in various drop-down lists, etc, would ideally be populated from data that the business owners can update. But somehow, the idea of "hard-coding is bad" got misunderstood. One former developer obviously missed the point... that the idea was to not require code changes when data changed. So we have a database full of hundreds of configuration documents. When you put these documents into edit mode, a big red warning pops up. "Do NOT change these documents without checking with Notes Development staff -- code changes will be required!!" Ehh... what??
So I look at the code. While it is true that these documents populate drop-down lists, the agents that then process documents are full of hard-coded business logic... "If docmuent.FieldA(0) = "Data from config doc" Then..." So if the data changes, the agents all break. Not only does any data change require a code change, but we are actually inviting the business to break things, and doing so in a way we need to debug code instead of just updating a field.
This is done all through our entire Notes platform. To the point that the business community here think that these kinds of configuration documents are a built-in feature of Notes, and all Notes apps will always work this way. In an environment like this, is it any wonder that people think Notes is a hopeless platform? I have a strong suspicion that if the Notes system had been well built from the start, SharePoint would never even have been a consideration here.
One of my current tasks is to refactor our SharePoint portal's design. The original design was heavily laden with tables and large images. Each new subsite had its own master page, style sheet, and images directory. It was performing poorly in the browsers, and making it painful to create new subsites.
So I'm making a CSS-based version, removing all the images, simplifying the design while improving performance and flexibility. The end result was much faster, but the look was too simple. It needed just a little something to give the design some depth. And that is where jQuery came in -- I picked up a Drop Shadow plug-in, and put some shadows around the main page elements. It was just enough to make the page look pretty decent. BUT -- performance degraded exponentially every time I worked with web parts. I haven't determined exactly where or why, but I do know that somehow the jQuery calls combined with what SharePoint does to a web part page in edit mode was very unhappy. As-in, browsers hang and crashes almost every time I worked with a page. So, I turned off the jQuery when a web part page is in edit mode by just popping the following into the code: if ($(".ms-WPAddButton").length == 0){ ...Do the jQuery Stuff... } In other words, I use jQuery to look for the 'Add a Web Part' buttons. If they are not found, I continue with the script. Editable pages have the simple interface, and run quickly. Normal content pages have the nicer interface... and still run quickly. There may be a better way to handle this, but it has worked for me so far.
Before I left on my vacation two weeks ago, I was starting to get more inspired about doing more with this blog. I joined twitter, I starting putting together technical content to post... But I was also getting tired of SharePoint, as previous posts show. I wanted to take my week offline and see if that refreshed me.
It did the opposite. I'm less inspired by SharePoint than I ever was. So this blog is complete. I think there is enough material here that it has served it purpose. It really doesn't add anything to my personal or professional life, so here is one big final post to close things out. I have heard from some people recently regarding my Domino work... I'll get back in touch with you folk via email when the time is right. Not that you will know it is me. :)
Was/Is SharePoint a success?
Partially. We had many issues when we first rolled out our portals. The Server farms were badly implemented, the UI designs were horrid, they were thrown together badly incurring massive technical debt, and massive maintenance costs. That is where I came in. Since then, we have restructured the farm, and while that effort still has many tasks before complete, we are stable with a plan to get to our ideal infrastructure within the next 6 months. We have refactored the UI designs, and their tehcnical implementation, leaving the content publishers with as much flexibility and power as any content management system. Our collaboration areas have grown, and our management of them has evolved, so we are in good shape for oganizational collaboration. However, the pains at the beginning mean that adoption of the technology was below expectations. Some groups gain much benefit from SharePoint, others ignore it or are downright annoyed by it. It was not the business-changing technology that was hoped for, nor did it save costs.
What would I do differently if I could go back and start from scratch.
1) Don't use SharePoint. Sorry, but the cost of implementation and support is high, and I think even a perfect implementation would offer a poor ROI. But ignoring that...
2) Don't skimp on technical planning. Think very carefully about long-term maintenance of the platform, and dig deep when analyzing the impacts of your decisions. Make every manager do a google search on "technical debt", and spend at least 30 minutes following the results.
3) Don't hire consultants -- at least, not when we did, when MOSS 2007 was so new. At the time of our launch, nobody was a MOSS 2007 expert. So why pay consultant rates for people who basically are using us to learn a new technology, and then just writing .NET code anyway. Instead, hire some really good, sharp people internally, and let them run with it.
4) Don't fall behind on patches. Ever. Microsoft knows that SharePoint has issues, and actively tries to fix them. If you are going to marry yourself to Microsoft, make it a productive relationship, getting their latest patches and fixes, and keeping in communication with them for support and to know their plans.
5) Train your users. Repeatedly. Record the trainings. Make them available online. The more powerful your users are, the more time the IT staff can spend on the platform itself, or on custom code. But that power needs to come with knowledge and insights that only training, experience, and a solid support staff can provide. Put in the effort to give your community those things...
6) Do not accept mediocrity in your IT staff. We all know that the corporate world has more than its share of mediocre IT staff. There are reasons for this, and it isn't a bad thing in many cases -- Not everyone can be a superstar, and much of corporate IT doesn't need superstars. But SharePoint does. At least as of today - I have hopes that in 3-5 years, the platform will mature into something more friendly, stable, and robust. But for now, it is plagued with problems. Any IT staff member can handle these problems in the short term. But it takes top talent to do so quickly, repeatedly, and often, for months and years on end without running into morale issues. Running a SharePoint implementation is the 100-mile run of IT. Sure, it can be done. But the people who can do it well are freaking machines, not your average IT guy. Get those people on board Day 1 if you want SharePoint to truly improve your business.
7) Code Reviews and Architecture reviews. Do them. Mistakes in design become nightmares in support. Trust No One.
So where does our Notes Migration Stand?
Over the last 18 months, I reduced our Lotus Notes environment from 6000 applications to just over 150. 35 of those are group mailboxes that are in the process of moving to Exchange. 50 of those are basic logs of what happens in our business -- the SharePoint replacement is in a pilot program, and they should be gone in a matter of weeks. 50 more are minor apps, and would only take about 3-6 months to move, but we put that effort on hold due to the recession and its impact on our business. Our 5-10 major, massive apps are slated to be replaced by 3rd party products. Again, plans are in place, but the recession has those projects on hold. In short, we expect to have moderate usage of around 50-65 apps for 2009, with everything else being shut off with a reasonable amount of effort in 2010 or 2011, depending on just how long this recession lasts.
The migration wasn't rocket science. A lot of it was just being willing to dig in and do the work. It does require an understanding of the business, the usage of the existing platform. It also takes effective communication and negotiation skills. A consultant would not succeed by walking in the door, getting specs, and making plans nearly as well as a long-time business analyst who understands not only what goes on, but why. The migration requires less technical skill, and more business comprehension.
Recession?
Honestly, without the recession, I'd probably keep going with the blog, the migration, and probably enjoy it more. But being shut down into a maintenance-only mode means that there won't be much to share anyway. Time to move my personal efforts into other projects...
SharePoint Conclusions
1) If you are already a solid Microsoft shop... sure, go for SharePoint. You probably have the talent to make it work. If not, don't move to a Microsoft-based platform just for SharePoint. There are other alternatives that will serve you better.
2) Don't buy into the marketing. SharePoint is an effective collaboration tool. But nothing more. Use its strengths, but don't expect it to transform your business in any way.
3) Everyone should revisit their stance on SharePoint in 3-5 years. Us Notes folk benefitted from the early leadership of Ray Ozzie. I believe SharePoint will, too. If you already run it, look for future enhancements. If you choose not to run it, don't let today's SharePoint deter you from the SharePoint of tomorrow. I think it may come together.
Lotus Notes Conclusions
If you've got a Notes/Domino platform running, stick with it. If you don't like it for some reason, fix the problems with your implementation of Notes. Don't throw it out, expecting a new platform to be better.
Personal Conclusions
1) SharePoint isn't fun.
2) SharePoint pays well. You earn every penny. Don't go into it for the money. Be a fan of the Microsoft Kool-Aid or it will slowly incinerate you from the inside out.
3) The breadth of my IT experience serves me very well. I've been a programmer, and analyst, a trainer, a helpdesk grunt, a server admin, a technical writer, a web designer, and a manager. I have used every single thing that I learned in all of those disciplines over the last 18 months.
5) I have learned SharePoint. I have a solid bullet point on my resume and in my skill set. I don't really care. Kinda like my Notes admin skills -- sure, I can do the job, but I'll likely never apply for that job. The SharePoint work has done well for my resume, salary, job stability, and all those things that I really should care about. But I'm a programmer at heart - none of that means much to me when the platform itself is dreary. Don't get me wrong -- I fully appreciate being blessed with a stable job at this point in time. But inspiring, it is not.
6) The past 18 months have been frustrating and educational, but with moments of great satisfaction when we finally do get things running the way we want.
7) In short, I'm getting too old for this crap. :)
8) Nevertheless, the closure of this blog changes nothing for me personally. I have the same job, the same roles, and the migration and SharePoint work will continue in my day-to-day life.
Final Words...
I intend to leave this blog up, as it does get search traffic, and I hope my old content can help some people out. The gmail address is forwarding to my personal address. If anyone has reason to get in touch, feel free... I'll be reachable in that way for the foreseeable future. Thanks for reading and participating in the blog...
Just in case anyone still has me in their RSS readers, a quick FYI: We just hit a nice milestone. We have under 100 apps left in our Notes Environment. As of right now, there are 99 apps remaining. 99 apps in our Notes Environment, 99 apps in Notes... Shut one down, archive it down...
IBM did come in and meet with us, trying to show us all the benefits of sticking with Notes, and all the new features in 8.5. Sadly, they missed the point. They showed us all the great new dev tools (xpages, udpated designer, etc), but nothing that would improve our business. It felt more like they were trying to recruit me into being their evangelist than actually listening to our needs. Then, in regards to the migration, they said, "We are not aware of ANYONE that has actually shut down their Notes environment." Well, that just sounds like a gauntlet being laid down. I doubled my efforts to shut the thing down. And we now have plans in place for all of our major apps. Those plans will take time to realize due to the economy, but we have an answer for every remaining app.
I know, I know... I am done with this blog. No need for any more updates. But I'm also feeling guilty that my last post was probably unfair to the IBM folk. Their attitude just came across badly, and set us against them. So I thought I would provide a little more info on how things are going...
We're down to 75 apps. All mail is finally off the system. Apps still send outgoing emails, but we have no more incoming emails. Room reservations are finally gone. I expect some people will be surprised that they stayed on Notes so long... the truth is, Exchange couldn't compete with the flexibility of the Notes reservations system. We had a customized system that would allow people to order different table/chair configurations for conference rooms. The only way we got off of Notes for meeting reservations was to force it via management.... just telling people that they were losing functionality and had to deal with it.
Another surprise - our two problem apps which really havent found a good home on the Microsoft platform may have a better answer. People are looking into force.com as a new platform for these apps. I haven't seen enough of it to comment on its technical feasibility, but the business folk like what they have seen, and one of our architects has confirmed that it can do what we need. I don't see how that solution will save money, nor will it simplify our environment. But if this migration was based in ROI and simplicity, it never would have started in the first place.
But I really wanted to share how the job is feeling now:
On the good side, I like this place that I work. It is a decent company, with good people, fair compensation, and reasonable management. It is stable, despite the effects of the recession. I have no real complaints about the company or my position within it. (Although it would be nice if they had a blog-friendly attitude, so I didn't have to fear for my job should I be outed.)
On the bad side, if I don't want to be a Microsoft coder (and I don't), I don't know what my future here holds. I've chatted with my boss about my desire to find a new path... so maybe we will come up with something. But one thing the last few years have taught me -- I just don't like working with the Microsoft tools. I never intend to take another job at a Microsoft-based shop. I intend to find another role within my current company at some point in the future. If I cannot, then I will need to consider a more drastic change. I've given MS a chance - I've spent 5 years learning their platforms and working with them. After that time, I think it is fair to pass judgment -- I am adamant that I do not intend to build a microsoft-based career.
I have two items that seem significant enough to be worth another post:
1) Migration Tools -- I stated in my last post that I was going to work on some migration tools. I designed what I thought a nice quite of tools would be , but then something interesting happened. I had reason to go look at the tools created by Binary Tree, which I had never actually researched until now. I have not actually used the tools, but I have read the descriptions on the tools on their web site. It sounds like their tools are a very close match to what I would have designed. This, along with some conversations with people in the community, makes me question whether it is worth doing anything on my own. If the tools already exist, I'm not sure I am doing the community any favors by making new versions, for which I would not be able to dedicate the same level of effort as they already have done.
2) Some recent "Goodbye, Lotus..." blog posts. I've recently noticed a few "Goodbye" posts. And I want to make something very clear - just because someone starts work on Microsoft platforms (Or Google Apps, or whatever), that does NOT mean that your connection to the Lotus world ends. I admit that I will now sometimes go a few weeks without reading Lotus-related blogs. But I still do care about it. I still put 15 years into my career as a Notes guy, and even in the SharePoint world, it gives me a different perspective that allows me to design apps better than someone who has only ever tasted the Microsoft Kool-Aid. Some of the comments and posts I have seen have acted like moving to SharePoint is the equivalent of a bad break-up -- time to move on, embrace the new direction, and never look back. I think that attitude does a dis-service to both the individuals who think that way, and the community as a whole. What has made the Notes community an interesting place is the people, the attitudes, and the differing perspectives from which we all come. It seems like few of us came from "traditional" programming backgrounds. Many of us learned on our own, and due to the nature of Notes, we often learned good practices in software design before we ever learned good practices in programming. This makes us unique in the IT world, as most other folk learned 'good' code first, then later learned how to apply that to good software designs. This difference in our perspectives can make us valuable software professionals under any platform. So I suggest that we all embrace this. No matter what underlying technology we end up working with, embrace what we have learned, and design good apps. If we do that, we may even dare to dream that someday we can make new platforms into the same positive, efficient and effective platform that we all know Notes can be.
So, I may have reason to re-kindle this blog. Let me update anyone who is still reading -- Since my last post, in which my frustration level at my work was hitting the breaking post, I have done a few things:
I also have finally sat down and started taking all my old scripts, enhancing them, and coding the toolkits that I feel everyone needs to manage a migration/conversion effort. I'm finalizing details, but I will most likely be partnered with some consulting firms to create these tools. Once I know for sure what the business relationships will be, I'll figure out what I can/can not post. But I'd like to get back to sharing my ideas and experiences with the community.
One of the topics I wanted to discuss was the differences between three time periods:
As a disclaimer, I don't really consider myself part of the "Notes Community". I read blogs, I write this one, and I know a few people. I have been doing Notes/Domino work since 1994. But I have never partaken in Lotusphere, have never really tried to mesh with a lot of you who are out there. So my perspective is as someone who hides in my corner of the world, working and watching. Other people may have different experiences.
5 Years ago, there was a bit of a resurgence in Notes going on. We had gotten through the "Websphere doldrums", and everything appeared to be on the upswing. People were getting excited about new versions of Notes, IBM was making a significant effort to bring the UI to modern standards, etc. Blogs were active, and people were waving their Notes flags high, proud to be Notes folk. Many people were still 100% Notes in their work. Migrating off of Notes was discussed at all levels of IT, but still was not all that common. My recommendations to people were to go ahead and continue to build out Notes environments... it looked like it had a future.
2 Years ago, morale in the community was pretty poor. I recall a number of people writing emotional posts about walking away from the Notes world. Most of the really sharp people I know had already picked up new skills, and were no longer spending all of their time in Notes/Domino. Notes job blogs popped up, which displayed most of the Notes jobs on any given day. Isn't that scary? When all the jobs can fit in a single blog? Despite the downturn in the community, some people still are fighting for Notes with all their heart. IBM does not appear to be among them, though. The new versions are getting pitched to developers, not IT leadership. The whole "Bleed Yellow" concept is no longer very strong...
Today, I know very few Notes people who still do it as 100% of their job. I know nobody who is fully committed to the platform. IBM's stance on things seem very nebulous... more buzzword heavy than technology-driven. I hear more from Ed Brill about his travel plans than I ever do about Domino. Consulting firms, and especially their sales folk, tell me things like, "In 2 years, there won't be a single Notes client left in the USA." Not only are migrations commonplace, they are somewhat of a solved problem. Multiple vendors compete to sell their tools to everyone who is shutting down Notes. I know a lot more people who skipped the last Lotusphere than who went. And I don't see a lot of excitement anymore.
I no longer recommend Notes. At All. Not because of any lack in the platform, but because the talent has moved on. Even if you were to hire someone to build the best Notes system on the planet, people smart enough to do it right have moved on the new technologies. Notes.net has been replaced by StackOverflow. The old Sandbox is deleted. (Fortunately, also archived and its functions replace by openNTF.) At this point, I wish IBM would just open source the whole platform. There are people out there who would roll with it, and it might bring some excitement back.
Bottom Line -- We're in a very different world now. While I know some Notes shop are still out there, and going strong, they are a minority. This difference in the market is going to drive my future direction for the next little while.
We cannot look back to where Notes was in its heydey. We can quietly hope that IBM pulls a rabbit out of their hat and recovers the market, but I cannot base a career on that hope.
What I can do is try to help people move on effectively. Because no matter how many questionable Notes apps I've seen over the years, nothing compares to the nightmare that is a bad SharePoint implementation. Notes gives you a lot of room for error, and is very easy to refactor and recover from mistakes. But SharePoint will beat you up, knock you down, and keep kicking you until you run home in a burned-out heap.
So that is my focus for the next few months, as I work on turning all my thoughts, ideas, documents, and code from the last 5 years, into a single tangible product. I hope to step people through this process in a simple way, driving them to a clean, maintainable environment that will bring them as much business value as Notes did, hopefully at a comparable cost.
(BTW, before someone asks, "What about XPages?"... hold that thought. I'll write those thoughts up tomorrow.)
I have looked at XPages a number of times, intending to learn all about it. This is driven partially by a desire to be at the top of the game when it comes to Domino development, and partially because I am seeing contracts that require XPages experience.
But I have yet to truly be sold on their value, with one caveat - if you are not a Web developer, XPages seem like a great way to develop Web apps anyway.
Let me put it in my perspective - when Google maps first came out, their use of AJAX was truly innovative. I recall people around the office crowing around a desk and saying, "Look at this!" And that day, I truly dedicated myself to learning web development. I always could do HTML and CSS, and some basic JavaScript, but I wanted to do more.
So you throw in a few years of strong effort to learn JavaScript, CSS, HTML. Then throw in jQuery, which doesn't actually build new functions into the browser, but makes it a whole lot easier and faster. You end up being a developer who can really make the browser dance, and who can work with almost any back-end system to develop rich browser UIs. All you need is a URL to point your AJAX calls to, and JSON or XML to parse on the return. Even in Domino apps, views and agents become data feeds to JavaScript functions. Forms become places to hold code, not much more. Domino still offer fantastic value as a RAD platform, as the flexibility of its data store is unmatched. But as an experience Web guy, it starts be less of an application in and of itself, and more of a framework to match up the flexible data store with your browser-side scripts.
So now, with that perspective, look at XPages. Really look hard. What does it offer that is above and beyond what I can do on my own? Not much that I have seen. And ultimately, that is why I have never picked it up or used it. I am fully functional on web apps without it. YMMV.
To me, it becomes yet another long-term management issue with developer talent. If an app is built in XPages, only XPages developers can support it. If it is done using more standard tools, I can take any front-end developer off the street and have them understand a large chunk of the code even if they have never heard of Domino. The learning curve to teach someone the IDE and Domino's data store is much lower than also having to teach them XPages.
I have a very hard time convincing myself to invest my time into a technology that puts limits on who I can call on for assistance, without giving me any tangible benefits above and beyond what I can do without it.
All that being said, XPages do serve one VERY valuable purpose -- Web-enablement of existing Notes apps. Most of the migrations I work on are looking for an easy way to lower the costs of the Notes environment. One common idea is tossing the client out the window, and just using a browser for everything. This is where XPages can shine. Most of the development work is already done. The goal is not to build the most modern web UI, but simply to get the functionality into the web browser. XPages can bring it all together more nicely than just the default results Domino will serve, and can be done quicker than custom web code.
TLDR - Xpages for New Development? Not a chance. But for re-factoring old apps? Sure.
A quick note of thanks to those who gave me more info on XPages in the comments from the prior post. I'll be checking out the links provided, and who knows? The points made seem pretty solid, so maybe I will change my tune? We'll see...
Being heads-down on migrations, I do not always keep completely up-to-date on the latest and greatest news on what is coming in Domino. So I did not know until last night that IBM is bringing forward a browser plug-in to let Notes environments run in the web, no client required: http://www.edbrill.com/ebrill/edbrill.nsf/dx/first-beta-of-notesdomino-social-edition-now-out-to-design-partners?opendocument&comments#anc1 This is significant -- Almost every customer I have worked for desires a quick, cheap, easy way to just push everything to the web. If this plug-in works as advertised, they now have that option. For some of my customers, this even removes a large chunk of their motivation to get off of Notes. I've talked with quite a few IT Directors/VPs who would be satisfied with a conversion to web, and then leaving Domino in place as a long-term application platform.
And if Domino gets back "in" to IT Architecture roadmaps, then that also bring XPages back "in" as a relevant technology. But, it removes the value in conversions to XPages, and now make XPages worthwhile for new development or refactoring. So maybe my prior post on XPages was short-sighted after all.
But as I mentioned in the comments in response to Nathan, it was an honest opinion. And one shared by other Domino developers that I know. I'm glad that I posted it, as it brought forward some great points in the comments, which are going to make me do more research and re-evaluate things.
The key question for me now is the age-old question that has plagued our corner of the world since 2002 - is IBM truly committed to Domino as a long-term strategic product? If so, maybe the advice to hire Java guys, and focus on Java-based development fronted by XPages makes the most sense. It keeps your existing investment in place, builds for the future of the Domino platform, and gives you a safety net, in that if IBM does bail on the platform, you may be able to transfer a chunk of the code to another Java-based framework. Having not yet seen it in practice, and not knowing its full functionality (or stability), this is all purely speculation. But there is a ton of potential here for significant change to how Architects will approach the platform.
I have done some more reading on XPages, and I am seeing a lot of great examples of how to put together simple apps quickly and easily. Lots of tutorials, lots of documentation.
Which is all fine and dandy, but I am hungering for something with a little more meat. What kinds of apps drive developers out of the built-in features and start pushing you to more in-depth JavaScript, or Java. How does a highly complex app come together, and how does it compare to the Microsoft stack? Not just SharePoint, but if you scaffold up an MVC 3 app, and put 200 hours of work into a complex solution, how do the technologies compare?
In my experience, highly complex apps build their own structure, over time. As you start, you may draw out forms, views, navigation in broad strokes, as a mock-up of where the app is going. But over time, you add in the details, and you start to build common functions to handle the details of your application. Over 2-3 years, you have often re-factored it to the point where the appsis its own framework. You end up with your own sets of quick functions for AJAX calls, validation, UI, security, business logic, etc.
So it begs the question - for the sake of argument, lets grant that XPages gives you a running start vs. "traditional" Web development. Does it maintain that advantage over an app that takes in 200 hours of development time? 2000? Or do all applications eventually reach a level of maturity wherein they all balance out?
Also, there is a more important question - does it matter?
At the end of the day, how long a developer worked on an application is frequently meaningless. The real question is whether the resulting app brings value to their business. Let's suppose an application brings/save a company $1,000,000 dollars per year. And suppose it took 400 hours to build using "traditional" methods. Suppose you paid $100/hour for that work. So you spent $40,000 to make a million.
So now let's say that XPages can do it in 10% of the time (which is an extreme example, to be sure). You spend only $4,000 dollars to make a million.
In both cases, the business made over $950,000. So do we really care about the development cost? Well, sure, every dollar counts, but really our primary concern should be the recurring costs that will eat into that $950,000 quarter after quarter-- server licenses, infrastructure costs, support costs, ongoing maintenance, training costs, etc.
So I'm inviting more comments - does anyone have real-world data to show how XPages performs in terms of year-to-year TCO vs. "traditional" Domino development, .NET development, and/or open source toolkits?
Has anyone done enough analysis to draw a line in the sand to tell us at which point an app is small and simple enough that the head start XPages may offer will provide a tangible ROI?
Also, can anyone put numbers to the claims that XPages performs better than "traditional" Domino apps? I do not doubt the claim, but are we talking a 1% improvement or a 50% improvement? Will it impact my TCO?
Don't get me wrong - my reading this morning has earned XPages more respect from me. But I need to know true business value before I will stake my career on a technology. I'm not sold yet.
I continue to find myself at odds with many developers. This is true for many platforms, not just Notes/Domino. And I have been thinking about the underlying reasons for this.
I have concluded that I am not a developer. I have the technical skills to write code as well as anyone, but I care more about why we are writing software than how we are writing it. I care more about business improvement and business value than I do about developer efficiency. I care more about strategic placement of technology within an organization than I do about the technology itself.
Even when it comes to technology specific questions, I think at more of an architecture level than a platform level. I care that we are deploying technology for the right reasons. I want to call out the underlying business problem we are trying to fix, and decide which technology has the best answer. I then want to determine if that "best answer" fits with the existing infrastructure or not. And if not, I want hard data showing me a cost analysis of which is better -- to go with a sub-optimal choice that fits the infrastructure, or to change the infrastructure to meet the best choice. I then want to balance that hard data with long-term strategy and understand how a choice today will impact the choices in the next 3-5 years.
It comes down to what drives my personality. I used to think that I loved to code because I love problem-solving. But the truth is that I am more OCD than that. I love problem-solving because I hate "clutter" in my life. I hate having backlogged things to do. I hate having business inefficiencies. So I am driven to make things more efficient, to resolve problems quickly, and to dig to the root cause of problems to ensure that they do not re-occur. I like to streamline an organization, make it run like clockwork, then sit back, relax, and let it run.
When I was younger, software development satisfied those urges. It no longer does. I try to get passionate about geeky SW Dev issues, but I just cannot do it. I can get passionate about the process around dev, making it better, getting the business results faster. I love watching the startup world to see what the young punks try, what works, what fails, etc. I think most internet startups are useless, if not downright harmful to our culture, but that is another story - whatever their social worth, they make a great testbed for new ideas. And that does excite me.
Also, when looking at new jobs. I no longer care so much about the job itself. I care about who I will work with. I care about the leadership. I care whether or not they have a technical clue, but I also care about whether or not they listen to their staff. I care about their actual leadership ability, and their level of self-awareness, to know when they reach their own limits and need to seek help. I care about how they seek that help, and how their actions impact the morale of their organization.
Oddly, I care less about their actual success than you might think. I have had great, positive experiences with companies that were utter failures. I have had horrid, negative experiences at companies that succeeded. I'd rather fail with a great leader and great peers than succeed while surrounded by mediocrity.
I'm not sure that any of this matters to the world at large. But it makes a big difference to me as I sit here, between contracts, trying to decide what I want to do next.
While I continue to wait on sign-offs on new projects, to know exactly where I am going next, I've been writing up my toolkit to assist people in running Lotus Notes to SharePoint conversion projects.
I have completed what can be called a 'rough draft' of the full toolkit. I have a laundry list of items to add to it, not the least of which is security, so it is not production-ready, but it is 'content-complete'. I intend to spend this week hardening it so it is closer to being production ready, as well as working on UI / HCI issues.
At this point, I'm happy to demo to friends, or people I know in-person, but I'm not ready for a public demo.
I am however, starting to send out email inquiries to see if anyone wants to pilot test this thing. I'm trying to design it to be useful either for a consulting firm running a client engagement, but also for an organization who is doing the migration on their own.
My hope is that this week is an eventful week. In an ideal world, I'd finish the first version of the tool, get a final decision on my next job, and find pilot testers.
I know, I know. We should be putting out professional content, focusing on improving the software development industry and showcasing the deep insights we all bring to the table. But sometimes... sometimes, it is just more fun to be snarky. And to write up content that is tongue-in-cheek, with questionable technical accuracy, and questionable taste. So this point falls within that spirit - snarky, possibly going so far as to be considered trolling, but still so much fun to write and post:
The Timeline of Xpages from 2002 To 2014
2002: IBM sez: All you Domino Folk, go become Java folk and do Websphere! Domino folk sez: NO WAY! You suck!
2004: IBM sez: OK, OK. Domino can stick around. Domino folk sez: WOOT! IBM secretly goes out and buys a Java-based toolkit that they intend to move all the Notes folk towards. SHH.
2008: IBM sez: How about XPages? Domino folk sez: Eh? Cool! Let's try it.
2012: Domino folks sez: XPages really is best when you write it all in Java. IBM sez: *snicker* Gotcha.
2014: IBM cancels all future development on Lotus Notes / Domino, stating that all the development in recent years has been done in Java anyway.
I have a track record of being able to fairly accurate see where technology is leading us. I vividly recall showing my father the Web, back in its infancy in 1993, and trying to explain to him that it was going to change the world. He believed none of it.
I also recall conversations with co-workers in 1999, in which I explained that the web was going to grow beyond simple content authoring and consumption, and start to become a platform for business applications. The extent to which that has occurred has surprised even me, as the whole smartphone thing and "There's an app for that" exploded beyond anything I had imagined.
More recently, though, I've been putting serious thought into where software development in general is going. And my conclusions are a big part of the reason that I have proven resistant to buying in to XPages. The problem is that business has spent the last decade or more working with big software vendors. IBM, Microsoft, Sun, Oracle, SAP, etc.
I am not sure those big vendors have as much of a future as they all hope. The development of cloud-based computing and open source platforms has to a point that a business no longer needs the stability of a large vendor to maintain the stability of their own operations. I am not saying that business should move to "the cloud", but I am saying that commodity servers running open source platforms have broad enough support, including professional technical support, that a business can now rely on such things.
This isn't anything new -- I always hear open source fanboys saying things like "Why pay tens or hundreds of thousands of dollars in licensing and maintenance fees to large vendors, when you can get the same reliability from <insert favorite open source platform here>?"
But they are starting to be correct. The broad technical talent base on open source platforms is an order of magnitude larger than it was 5 years ago. And more competent. Many of the people who rolled out of school, into the internet startup world, now have actual experience under their belts. Also, the number of people who understand enterprise IT as well as open source increases every day.
Likewise, there is a new dynamic in town as to how people get experience. In my time, you got a tech support role first, learned how IT runs, then slowly built up to being a developer, where you can really have a strong impact. Now, a lot of young folk go straight into startup work, develop something cool, have it crash and burn on them, are forced to learn business, and still end up having a clue by the time they hit 30. And all their pain and failure was on their own dime/time, too, so it is often taken to heart a little better.
I'm kind of rambling here, but the point is that we used to work in a world where you stepped in as a cog in a machine, and fought your way up to a position of importance, but had a firm grounding in "How things work". That world rejected open source as too flighty, not stable or secure enough.
But the new order of things is for people to charge out with guns blazing, innovating before understanding, falling on their face, but walking off the field with some pretty solid, respectable talents. While this makes for a terrifying experience if you ever want to hire someone under the age of 25, it also has forced open source platforms to mature, as they are getting hit from all sides with ideas that are good, bad, brilliant, and horrible.
To try to get back to my original point, this new dynamic in the talent pool is going to start to serious dampen the success of large vendors. Why would someone who spent the first 10 years of their career doing small business or startups ever make the choice to call up IBM for help with anything? Or Oracle? Or even Microsoft? And what happens to the large vendors when those people grow up to be the leaders within the general business community?
The days of enterprise IT will never be over. Large business will always be around. But as people come up the ranks who do know how to make open source platforms sing and dance... the days of large, expensive vendor-owned products will diminish.
So my prediction is that as much work as I get today, moving Lotus Notes over to SharePoint, in 5 years, I'll start getting similar projects - shutting down SharePoint for some open source platform that isn't even developed yet.
I'm starting to get contracts that have no Domino work at all - just migrating from SP 2007 to SP2010. I have not yet needed to be a SP2010 expert, but now is the time to become one, so I can continue to offer value to my customers as they fall deeper into the pit of despair that is SharePoint.
In any case, one new feature of SP2010 that I probably should have known about before now -- In SP 2010, you can now mail-enable your lists and libraries, and accept incoming emails as new list/library items. This works on standard lists and libraries, not custom lists. Still, that is about the same as mail-in DBs in Notes, where the email just gets dumped into a Body field. And that is one feature that I now need to remove from my list of "Things Notes does that SharePoint does not."
Another SharePoint 2010 feature that I knew about, but did not think deeply enough was Document Sets.
In short, a document set is exactly what it sounds like - a set of documents. They have a shared set of metadata, as well as metadata on each document. The author of the set can control what metadata can be changed at the document level. Workflows can be run against individual documents, or the entire set.
From a migration standpoint, this sounds like it will cover many of the reasons that people use response documents. Not every scenario, to be sure. But when we see that basic function of records grouped under a parent record, this may be a viable feature to meet the underlying business drivers.
The demand for SharePoint talent right now is intense. More so than I have ever seen demand for anything in my career. I have already mentioned that I have been waiting on a few contracts to decide where to go next, but while that waiting has been going on, I have had 5 separate "offers" on the table. By which I mean discussions that have gotten to the point of, "We want you on the project", but I have not pursued it to the point of a formal contract.
And I am not even advertising that I am available.
On the other side of the table, the leading candidate for my next gig is also trying to hire a SharePoint Architect, just for a short 6 week engagement. And we cannot find one. The people who are really good at SharePoint find work almost overnight. And they are picky - you get folk like me who sit back and wait for something really good, not just jumping at any contract that comes in.
Every recruiter I talk to says the same thing - SharePoint talent is really hard to find. People who claim talent but aren't that good are easier to find, and most companies will take that, too.
At this point, if you can even do so much own a department level sub-site within SharePoint, you can find SharePoint work.
I'm not complaining - with our economy's current state, I'm thrilled that work is this easy to find. I just wonder how long this will last before the demand pulls in so many people that SharePoint skills become as common as HTML.
Ok, my long-awaited answers on my next project finally came through this week:
1) The migration tools/methodology project I've been kicking around since January got put on hold, for good reasons, as they realize it is bigger than a breadbox and want time to think out the long-term strategy vs. doing everything piecemeal. Good call. Maybe later in the summer. Maybe not. Depends on what direction the customer ultimately wants to take their business.
2) I am taking a 5 month migration project in Denver in the meantime. SP 2007 --> SP 2010 migration. Not a single bit of Lotus Notes/Domino involved. I may also do a short 2 week engagement to help a local municipality get their chaos under control before that migration starts. But for the foreseeable future, I am 100% SharePoint, 0% Lotus
.
So what about the toolkit I've been working on?
As a result of taking a full-time gig, at a real office, I am actively seeking out partners to drive my toolkit forward. I will not be able to dedicate full-time effort to it, so I cannot do it alone. I'm going to try to put together some demo / marketing material to show what it actually is, and what business value it offers both to consulting firms who may use it, and their customer base.
Really, I'm busy enough that I don't have much ego wrapped up in this. It is only a few weeks of effort. I am equally happy to just sell it as-is to a company who could run with it, as I am to find a partner.
What I would like from a partner is someone who has a sizable customer base with an interest to move from Domino to SharePoint. We'd pick one or two customers to pilot the tool as-is, and go through a few iterations of feedback on its usability and effectiveness, hopefully launching within 3 months with a well-tested version that has gone through real-world projects.
If anyone out there wants to partner with me on my efforts, or just buy the code outright for your very own, let me know...
I have been seeing new migration tools and companies pop up over the last couple years. And a lot of you seem to be focusing on automating a conversion of Notes/Domino applications to new platforms. I see marketing talking about how easy the tools are -- just apply to your existing app, it will rebuild forms! views! even business logic!
Now, I really do think that is valuable in some case. But it is a BAD, BAD idea to make this the basis of a migration project.
If a Notes application was working perfectly for the business, doing exactly what they needed, with perfect HCI, then why migrate at all? On the contrary, most Notes apps are old and outdated, and need changes to match today's business needs. The forms and views as they exist are driven by Notes conventions, not business needs.
And some of the biggest complaints about Notes/Domino apps are the UI and HCI.
So why in the world would you want to replicate the exact app structure that is failing for the business??
You will get the same garbage that you had in Notes, but you will get in on a platform that needs more support. And you need to train your users again! As a bonus, you get to discover a whole new set of bugs and problems with your new platform, and thereby decrease your end user satisfaction while increasing your IT frustration! Fun for the whole company!
Not that I think these tools are worthless - on the contrary, I think some of them look like brilliant pieces of work. But their business application is much narrower than their marketing would have you believe.
There are always one or two major applications within a Notes platform that have become large, monolithic apps, for which a re-write would be a 1-3 year project in and of itself, simply because of the business complexity. These apps tend to be the roadblocks that prevent a company from shutting down Notes. And therefore, these apps are the root cause of the entire cost of your Notes infrastructure.
It is for these apps, and these apps alone, that automated conversions make financial sense. It will not improve the business at all... on the contrary, it will probably be a mild detriment to convert these apps. But there will be huge ROI gains, as you can then actually shut down your Notes servers, get rid of the client, and save a boatload of cash from your IT budget.
So if that ROI is large enough to mitigate the problems of a "Garbage In, Garbage Out" approach to application migrations, and also cover the cost of the conversion tools, then yes... there is a business case to purchase and use these tools for a very select subset of your Notes applications.
But it would be tragic to carry out an entire migration using such an approach.
First, a post of interest -- http://www.rivit.ca/?p=178&option=com_wordpress&Itemid=528
Second, the rebuttal by Ed Brill:
Every week I talk to customers who have thousands of Notes applications where there would be no ROI to transition them to HTML5 or XPages. They are simple discussions or doc libraries or teamrooms or workflow apps that have been running for a decade. What would the ROI be of doing that kind of transition? On the other hand, there are strategic apps that are core to organizations that they should absolutely transition to XPages or HTML5. The browser plug-in gets this out of having to be an all-or-nothing proposition, just to get to web-based delivery.
Third, my reply: Ed seems to be hearing different conversations than I am. The quick, simple apps are the ones that are easiest to move to SharePoint, and that are most closely aligned with SharePoint's capabilities. His question of ROI may have been rhetorical, but there is an answer -- quick migrations to bring the business functionality into a cohesive web portal, while simplifying your desktop environment, and eventually shutting down Notes servers, and reducing licenses.
He is correct, though, that the large business-critical, behemoth apps are the concern. While he thinks they absolutely should go to HTML5 or XPages, that is where I am 100% opposed. These types of app do not gain their ROI through technology - they gain it through business process improvement. Nothing we do on the tech side will make a huge dent... instead, when the business is ready for a serious re-factoring of the process, THEN move it to a new platform. And at that point, a Microrsoft Solution isn't any more burdensome than XPages. I, therefore, promote the plug-in for large complex apps. Minimize your risk by moving the existing app to the web without major code changes or business refactoring. Simplify your infrastructure at the same time.
One other note -- I hear a lot of complaints that the browser plug-in only replaces the basic Notes client... So freaking what? Most of the shops I work with stopped doing any serious Notes development at R7, and have been floating along, waiting until they have the resources to migrate away. The whole concept of the "basic Notes Client" is meaningless to them. Developers are complaining because they cannot do all their cool tricks. Business leaders have no problem with that limitation.
I'm 2 days into a SharePoint 2007 --> SharePoint 2010 migration effort, and discovered a pretty slick jQuery script:
http://spservices.codeplex.com
It is a jQuery wrapper around the SharePoint Web Services. I've always seen people access SharePoint data via the .NET APIs or PowerShell, but that has its own administrative challenges.
It has always been possible to write AJAX calls to hit the web services, but this jQuery plug-in makes it a whole lot easier. I discovered it this morning, and by the end of the day, I have a working Javascript-based scanner to walk through the entire SharePoint environment at my customer's site, logging all the sites and lists, who owned them, and when they were last used.
There are plenty of other scripts and tools out there that do the same thing, but writing one from scratch requires some fairly decent SharePoint knowledge. The reason I am so excited about this plug-in is that it lowers that knowledge barrier, and brings some of the power of SharePoint into the hands of modern web developers.
Almost more intriguing, it opens the possibility of easily using SharePoint data as a data source in HTML5 Web Apps.
I'm going to have to play with these ideas more as I have time, so stay tuned...
I have spent the past week putting together an inventory of a long-established SharePoint environment, and realized that the support experiences are diametrically opposed.
In a typical Notes/Domino environment, there are a few major apps that take up a lot of your time, some established processes to handle users, IDs, access, etc, and a plethora of small apps that get built, then just run for years without much support. So when it comes time to migrate away, the process of getting your DB inventory is 80% finding all the small stuff that the IT support team doesn't deal with very much.
SharePoint seems to be the opposite. There are so many complexities and so many different ways to do things that the IT support team takes a large number of calls on all the small sites, answering how-to questions, doing basic support on new lists and libraries, tweaking workflows, etc.
And when we go to inventory everything that is going on in their environment, we discover huge business-critical processes that have crept up without anybody even noticing, and that require no support because the department that built them did it all on their own and has their own guy maintaining it.
There was a time, long ago, when Notes was like this. In the early 90s, pretty much. There were even certifications for Power Users, who would go do a lot of the small stuff in their area. Many of us old-timers in the Notes world started as a Power User, and have learned to code "for real" over the past 20 years, as the needs of our Notes environments push us to deeper levels of technology.
It seems to me that Notes development centralized within IT for two reasons:
1) IBM bought Lotus. They managed Lotus internally in a centralized way, and that culture seemed to drift out to their customers as well. I worked internal to IBM from '94-'95, and again from '97-'99, and got to see their internal rollout and expansion of Notes, and it was very different than pre-IBM usage patterns.
2) All of us "amateurs" put enough years in to become professionals. I was a desktop support guy who took over the Notes environment, became a programmer/developer, and moved on from there. I know other people who were Power Users outside of IT, but moved into IT as their skills developed. Either way, the talent got centralized in IT, and then the culture and processes got centralized.
Why do I bring up this old history?
Because I think SharePoint must follow a similar path. As nifty as SharePoint is, the marketing hype is losing its edge, and people are realizing that it is a difficult, obnoxious platform to support, and the business case for running SharePoint is not as strong as everyone once thought. But it also seems like everyone has already installed it. Too late now. Oops.
So we have a lot of cowboy-filled environments, with the users going crazy over all the power they have, overstepping their skill sets, and running to IT desperate for help in fixing something that IT never saw in the first place. And though we all know the "right" answer is, "You built it, You fix it", that doesn't actually fly in reality so well. If the business relies on something, the business-need to keep running will override the IT-need to be right.
So the SharePoint consulting world right now is all about "Governance". It has been for a few years. Because the only way we are finding to control the environment is to lock things down. Take away the power, and run it through a "process". Really, Governance is a very formal way of trying to force SharePoint to the place Notes ended up anyway - centrally managed by IT.
I'm having trouble coming up with a final conclusion to this post. I noted the differences in support and just starting rambling from there. But I do think that those of us who have been through the full 20-year Notes lifecycle -- seeing its birth, sale to IBM, explosive growth, slowdown, and (arguable) demise -- we have a much broader perspective than we may realize. I recommend that we all take the time to step WAY back from the details of implementing SharePoint, and look strategically at where it may really go over the next 5-10 years, and what the long-term impacts it may have on our organizations...
Personally, I think it will not last 20 years. I think it will slow down in 3, migrations away from it will start in 5, and it will be gone in 10 years.
But I'd love to hear what everyone else thinks...
I've now seen 3 blog posts showing off the look of the "new" Notes email client. And these posts frustrate me, mostly because this does not look like a new, shiny design to me.
It isn't really a new design at all. It bears a very striking resemblance to IBM's intranet from the last time I was an internal IBM employee -- 1999. I really cannot be all that impressed by a UI design that is derived from a web page done 13 years ago.
Not that I dislike the design -- I think it appears to be an effective design. But there is no way I am going to start praising it as something new and revolutionary. I'd be much more inclined to call it what it is -- time-proven UI layouts and components that have been re-purposed for the email client. There is nothing wrong with that description and a lot to be said for going with a UI paradigm that has been well-worn for the last 15 years. Little risk, easy user acceptance, etc.
But let's not make it more than it is. It is new to Notes, and shiny... but old and non-original. And that is OK.
Microsoft Surface is all over teh interwebs today.
And all I can think about is the video:
http://www.youtube.com/watch?v=CZrr7AZ9nCY
I saw a link to the following site today: http://infolib.lotus.com/resources/experience/notes/
It is basically an ad for how great Notes is. And it is 100% focused on how to use email better.
Really? Because in most of my engagements, we are talking about how to get away from email. How to encourage discussions to happen on collaboration sites within SharePoint. How to use a universal task queue from K2 so people can manage their work from an Enterprise Portal, not their inbox. How to share documents online, not in email.
In short, how to bring an organization's culture beyond email, and become truly collaborative. But IBM apparently wants to keep marketing how great email is. And most of their "tips" show an underlying assumption that everyone works out of ther inbox all the time, and is constantly overwhelmed. I did not see one mention of the fact that Notes is an application platform, and not just email.
Hence my title -- Is IBM kidding? Is this a joke? Do they really think a few unique menu options in the email client will make people flock to their product line?
This is not the 1st time I've seen this either. In 2005(?) IBM had a presentation in Denver to announce the release of R7. Having just come off of a huge Microsoft Conference, with thousands of attendees, massively impressive presentations, and a vendor floor full of hundreds of booths, I knew IBM's presentation was going to be smaller. But it was underwhelming even with lowered expectations. 20 people showed up. IBM titled this presentation, "Getting out of the Inbox" or something like that. And then proceeded to give us the same old case study of a sales team, and how they were using Notes. And guess what? It was 100% focused on centralizing your life in your inbox.
A few of us looked at each other in disbelief, but gave it a couple of hours anyway. When we finally got sick of it and walked out, the presenter stopped his presentation to try to guilt us into staying. Again, we just kinda looked at each other and walked out. Hands down, it was the worst presentation I have ever attended, and I knew right then that IBM was WAY off track in their product lines. I had hoped that Notes may some day get back on track, because while SharePoint pays the bills, I don't like it. I do not like Microsoft's product lines at all. I like Notes. I really do. But it is not a viable career path. I guess I had hoped that in the past 7 years, IBM would have recovered.
Apparently not.
I posted a while back on SPServices, a jQuery plug-in that simplfies making calls to SharePoint's Web Services. I've been playing around with it, and have created a script that pulls all the data from a SharePoint list into an array of JSON objects (one for each list item). I then can iterate through the array, picking out the fields I want, and work with the data without any further AJAX calls to SharePoint.
As a proof of concept, I basically re-wrote the SharePoint View UI. I laid out a table of data with column filters on top, using jQuery.grep() to enact the filtering. There were some surprises involved -- first of all, a lot of the data was combined into a massive metadata field, which was difficult to parse. However, SPServices includes parameters that will break out this data into separate fields. SharePoint also tends to throw the datatype into the values, so I had field values showing as '#string;SomeString'. Again, an easy fix -- i just wrote a function to clean those values out. But there was a lot of trial and error involved.
Once I got it working, I was thrilled at its success, but also laughing at myself because I spent hours re-coding something that is already built-in with SharePoint. But -- it was just a test of the JSON data. mostly. My true intent is to combine it with Highcharts, allowing self-service dashboards against the data. I'm doing this all in preparation for a specific contract that I may work. Because there is a client involved, I do not feel right sharing the code at this point. But if anything falls through, and I do not work the contract, I will post my work-to-date.
The problem: The built-in navigation on SharePoint sites has 2 basic choices. You can let Sharepoint build dynamic navigation, showing the sub-sites and pages from your current site. Or you can manually create hard-coded navigation. SharePoint will create nested menus from its dynamic content. It will let you nest one level of navigation. But not two. There are a variety of solutions to this online, but they pretty much amount to the same thing: Go write some .NET.
My Solution: The same solution as every other day -- jQuery. The HTML created by SharePoint has no IDs, no names, and no distinguishing features at all, really, so the biggest trick to using jQuery to inject another layer of navigation is getting a handle on a DOM element to act upon in the first place. I ended up creating a placeholder item to be the 'parent' of my nested menu, and give it a unique URL.
I then can do this:
$(document).ready(function() {
var node = $('a[href$="/my/placeholder.aspx"]');
node.append("<ul class='dynamic'>" + "<li class='dynamic'><a class='dynamic menu-item' href='http://wherever.com'><span class='additional-background menu-item-text'>Menu Item Text</span></a></li>" + "</ul>");
}
);
To translate that little script into English:
To make your menu, you would just edit/add more <li>s. I placed this in our master page, so it is universal to the entire site. I briefly thought about reading from a list for the menu items, to allow maintenance of the menus without requiring a master page change, but I decided against it for 2 reasons:
I'll take an occasional maintenance effort over poor performance any day. This solution reproduces the same HTML that SharePoint would have produced if it allowed nested menus 2 levels deep. Which means it uses Sharepoint's CSS and matches everything else in your navigation.
Sorry for being offline for a month. Real life intervened. But I'm slowly getting back to online "work". I do still browse Notes blogs, via planetlotus.org, and through there have been watching http://mynewnotesblog.wordpress.com/.
Something has just felt "wrong" about his content recently, even aside from his heavy and obvious agenda to be pushing IBM's tools for social business. And this morning, I realized what it is -- Social features of software are not inherently beneficial to business.
I do not deny that collaborative software can improve communication and thereby improve business. Nor do I deny that IBM has learned a lot about tooling up to work with remote teams, as their workforce has dispersed geographically. But the big push to be "social" just lacks meaning for me. Partially it is just a buzzword, and each technology vendor labels different features as "social". But mostly because being social for its own sake is no better than technology for its own sake.
New software features need to have meaning. They need to return ROI, and their impact needs to be measurable. The chatter about social business online just feels like empty marketing to me. Again, I do not deny that used properly, a business can gain value through new technology. But the implication that some new software features will change a culture and make better business falls flat with me. Leadership makes those kinds of changes. Software may be a tool in that process, but software is just that - a tool.
I find myself working with InfoPath data, wanting to use jQuery to read the data published into a SharePoint list. But I have found that InfoPath creates internal field names in SharePoint that are somewhat unpredictable. It adds a zero at the end, sometimes, if you publish directly to more than one library, etc.
So, I wrote up the following little script just to spit out all the fields and values in a list, to give me a very quick view of the internal SharePoint names that were created when the InfoPath form was published. The data is produces is not pretty, but does give me what I need to be able to know exactly what SharePoint
Copy/paste everything below into a html file, adjust your urls and list names, and the path to SPServices, and it should work for ya.
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js" type="text/javascript"></script>
<script src="jquery.SPServices-0.7.1a.min.js" type="text/javascript"></script>
<script>
$(document).ready(function() {
$.support.cors = true; //for testing from a local file. remove when placed into a web part.
$().SPServices({
webURL: "http://your.site.com/subsite/",
operation: "GetListItems",
async: false,
CAMLViewFields: "<ViewFields Properties='True' />",
listName: "YourList",
completefunc: function (xData, Status) {
if (Status == 'error') {
$("#errorHolder").html(objToString(xData));
} else {
$(xData.responseXML).SPFilterNode("z:row").each(function() {
$.each(this.attributes, function(i, attrib){
var name = attrib.name;
var value = attrib.value;
$('#errorHolder').append(name + " | " + value + '<br>');
});
$('#errorHolder').append('<hr>');
});
}
}
});
});
</script>
<div id='errorHolder'></div>
<div id='debug'></div>
<table id='dtable' border=1 cellpadding=5></table>
I know that I have been quiet on here recently, and wanted to give a few updates:
My Current Project
Our SharePoint 2007 --> 2010 Migration is going poorly. There are just too many 'quirks' in SharePoint. Too many gotchas when moving content, and the tool kits that the client has chosen add gotchas of their own. It has taken longer than expected, been more painful than expected, and just overall been a fairly draining experience. I've said it before - working in SharePoint just isn't much fun.
Toolkit Status
On a related note, I have done very little on my migration toolkit. Between the project work, and an eventful personal life, the summer got away from me with little progress. I do have the toolkit in a stage that I would be comfortable piloting it to some people. I would be very comfortable using it as an internal tool for a consulting team. But I'm not comfortable letting it 'into the wild', and having unknown organizations use it without me. I do have confidence in it, but it just has not gotten enough testing. For some time now, I have been idly seeking a consulting firm to partner with who would like to use the toolkit, or a product firm who wants to integrate it with their own tools.
As a reminder to anyone who is interested, this is not a tool to actually perform a migration. It is a tool to analyze the environment, make recommendations and reports on the scope and complexity of a migration, and track the business of doing the migration -- identifying business ownership, sign-offs, migration plans, status, scoping, pricing, etc. Basically, it tries to wrap some of my experience into a web app to guide people through their migration.
I have also kicked around the idea of running this app on a SaaS model - For a first version, you would upload a .ntf of your DB design, I spit back a report on whether or not it is a good candidate for SharePoint, whether customization to SharePoint would be required, what the red flags are to be aware of, and a scope estimate to perform the migration. I'd add in the rest of the features later. I am not sure what the interest level for this would be, nor what a fair price point would be, but if anyone has input please leave a comment.
Blog Status
The original intent of this blog was Notes --> SharePoint migrations. Aside from the toolkit discussed above, I think this is now well-covered material online, and any additions I would add on that topic is becoming trivial. I have expanded to SharePoint-related topics in general, but I'm considering expanding even more, to IT in general, and technical leadership. My only hesitation is whether or not this blog is really a good home to a broader topic base.
I don't get a ton of readers on here - maybe a few dozen a day. So I'm not concerned with alienating my audience as much as I am about branding - in 3 years, do I want my personal blog to still be labelled 'migratenotes'
Again, opinions welcome in comments.
One nice thing about "Ye Olde Notes Worlde" is how simple the infrastructure really was. Install a server or twelve, maybe an add-in or three, and write your code. SharePoint is not like this. Every new bit of code is a new feature. Is it installed on your farm? Active in your web app? In your site collection? In your site? Want to use Excel Services? Visio Services? Great, another server install, more activation.Every piece of new code impacts your DR plan, and the process to add a new server to your farm.
This gets a little less painful with each new version of SharePoint, but there are also ever more new features to throw into the mix with each new version, so it is still a painful platform to maintain.
This came to light again recently when I started looking at ways to present BI dashboards. I was looking at PowerPivot as a potential solution, after seeing a demo at a user group. But then we started looking at what this really meant - a complex server install on SharePoint, with complex integration to SQL Server, which not only impacts our farm maintenance, but also impacts the SQL DBAs. We don't have a robust infrastructure team at this client. They simply don't have the resources to support additional complexity. So we have to eliminate this option for the time being. And by the time they will have the resources, the local Cognos team could probably do a better job anyway.
So at the end of the day, the decision on how to move forward with a dashboard project was decided by the infrastructure team. The reason this matters is that as a SharePoint developer, you need to know where your infrastructure team and your SharePoint admins actually stand. If you just grab every possible tool/toy available for SharePoint development, you can quickly become highly capable, but at the cost of complexity and additional effort for your admin team.
I've seen this turn adversarial at multiple clients, and you end up with an Admin team that wants you to do nothing but OOTB functionality (which is the policy at my current client due to their extremely limited resources). This admin team does not trust the development team, and feels that developers do not really understand the full impact of developing on SharePoint. And they are often correct.
I saw this happen at IBM in the early days of Notes as well. In 1998, there was an internal IBM conference post-Lotusphere, which included a small session to discuss what app dev best practices should be for internal IBM Notes apps. And the sysadmins took over. They came down hard, basically saying that all decisions need to be made by prioritizing the performance and stability of the servers. While there is obvious merit to stable servers, the general attitude in the room was that if one side of the fence had to accept compromise in their decisions, it was the developers. Over time, the Notes admins relaxed. They did not compromise the integrity of their environments, but they grew trust in the developers, seeing that the devs had a clue.
The SharePoint community needs to get there. Devs need to understand the impact on the infrastructure. They need to work with the admins to prove that we can develop solutions that do not complicate the maintenance of the farm. And doing so is not rocket science - just package your apps into features, keep DR/recovery documentation up to date, use source control, etc. This is nothing that a mature dev shop is not already doing. But if you cater to the admin team a bit, and build some mutual trust and respect, you'll not only work better together, but be able to expand the functionality of your environment together.
Why does the burden of making this work fall on the developers? Well, because we are the ones making changes. The purpose of development is to change the status quo. But we need to do so while being overly cooperative with all the teams who support us.
I ran into a blog today that has much more direct knowledge about the Notes Browser plug-in than was available a couple months back: https://www.ibm.com/developerworks/mydeveloperworks/blogs/rajpatil/?lang=en&CT=ISM0007 Two things to note:
The bottom line is that no matter what the Notes "blog-o-sphere" likes to say, I get a consistent story from the clients I work with. They have 2 big complaints about Notes -- Having to maintain the client install, and long-term maintenance of the application code. The maintenance is always the #1 complaint.
XPages does not solve those complaints. If anything, it makes those complaints worse, as you now are adding another dev skill set into the mix, and one that still has very little transfer into other platforms other than J2EE.
Like it or not, Notes is a legacy app. Companies are not migrating away from it because they don't "get it", or because of bad marketing. Although that may have been the root cause of the current trends when it all started 10 years ago, that was 10 YEARS AGO.
Today, the industry has moved in other directions. The talent has moved in other directions. IT leadership is no longer trying to decide if Notes is part of their long-term strategy. It isn't. But migrations also do not always have enough ROI to pursue. So Notes platforms stick around in maintenance mode for years. Which pretty much is the definition of a "legacy app." So, with that perspective in mind, that we are trying to have a cost-effective way to maintain a legacy platform... The Notes browser plug-in sure looks like a great option to investigate.
The current project I've been working on is to migrate about 1000 SharePoint sites from MOSS 2007, to SP 2010. And as of yesterday, the migration is done.
It feels good to hit this milestone. We are not done yet - there is a lot of cleanup work to do... shutting down old servers, etc. And the UI differences are making for increased support levels and complaints. Also, the functional differences between versions are painful...
Lotus (IBM) always worked very hard to maintain backwards compatibility. Microsoft puts zero apparent efforts into this. They remove functions, change limits, etc in fairly significant ways. To overcome all the changes, we first had to know what they were. We had to run a test migration, and sort through literally tens of thousands of errors in our logs to determine what was even going on. We then had to look at each site and make a business determination of which problems truly had to be fixed, and which ones were acceptable. Then we spent about a month working with the vendor who created the tools, reporting bugs, asking questions, upgrading versions, re-testing, etc., before we were finally ready to do the migration for real.
The funny part is that Microsoft says you can just move your content DB to a new server and upgrade it. And that may be true in simplistic environments.
Our experience was much more painful than that.
So as of now, I am spending 2 more months with this client, helping out on support, cleaning thing up, and building some BI functions. But I am available again in December... or maybe January, just to take some time off for the holidays.
Um. This Link--> http://www.bleedyellow.com/blogs/dotdomino/entry/the_free_lunch_is_over?lang=en_us
Well, that is certainly one perspective. But I disagree. Because I've been doing migrations for over 5 years now, and not once have I EVER seen a company that wants to migrate or modernize all their old, small, line of business apps. They are almost always old and outdated.
Business processes change, strategies evolve, re-orgs happen, people come and go. There is constant change in business. Those old apps are still in place because they are easy to maintain, and they do an OK job. Or sometimes they are just storing data for auditing purposes. They are not perfect matches for the exact needs of today... they are just acceptable tools for handling small jobs, not worth the trouble to change as the business evolves.
In my experience, the first step in any migration that I have been involved with is to get in touch with all the owners of those apps and simply ask, "Hey do you still need this? Does it still work for you?" I get about 40% of the responses back with, "No, just delete it." Maybe another 20% of the responses are "Can you just store the data somewhere", and 10%: "Oh, I left that job years ago. Ask someone else." (Which often means the app stopped getting used when job changes occurred.)
Of the 30% of the apps you really need to worry about, a majority of them (60%) are template-based: Calendars, discussions, etc. Or they are one-form apps. All of which are very easy to migrate to SharePoint, if not outright built-in.
Now there are some features of Notes/Domino apps that do not go over to SharePoint so easily. Of these, I'd say about 75% of them can still be done in SharePoint using built-in features, but they require someone who really knows SharePoint to do it properly, as they will be complex.
For those of you doing the math, that means that about 5% of your environment is truly going to be an issue that needs a new solution. And most of that 5% are probably apps that the IT staff already spends most of their support time on. As a matter of fact, when I conduct my client interviews at the start of a project, I always ask which apps need the most support, and there is a strong correlation between those answers and the "problem apps" when developing migration solutions.
There are also always a couple surprises - an app they thought was simple turns out to need a custom solution, or an app that was very complex in Notes is actually very easy in SharePoint. So the implication that is made in the linked post above, that there is this hidden elephant in the room of heaps of small apps that need to be acted upon... I just don't buy it. It does not match my experience.
What I have seen is that success comes when a business takes that 5%, takes some very large steps back, and asks, "What core business tasks are we really automating here? Can we do it better?" Often the apps can be simplified, re-designed to match today's business processes, and with a modernized UI all at the same time. The effort to do this is not trivial - it requires a dedicated project team to carry out the work, it requires communication and cooperation with the business users, and it will incurs costs. But at the end of the day, what does a business really want? An application environment that meets today's business needs? Or the same old app functions as they have been limping along with for 10 years, but now with a Shinier UI?
http://www.itnews.com.au/News/316955,bureau-of-statistics-retains-notes-in-it-transformation.aspx I've seen a number of blogs sharing their pleasure with this article this morning. Surprise, I also am pleased with this! People are focusing on the SharePoint comments, but I want to call out a different quote:
We have Notes programming skills in our team. They are very smart and very technologically literate
This matches what I have been saying for a while - the decision to stay with Notes or to migrate away is not really about the technology anymore. There are multiple technologies that can meet collaborative business needs. The decision is more strategic than that - one issue is always your staffing.
Whatever platform you choose, do you have the talent to use that platform well? These guys do, and that was a factor in their decision. Switching to a new platform needs to offer increased value to make up for the pain of migration. If you have enough talent to support your current platform, and the new platform doesn't offer compelling new business value, why would you switch? Now, to be fair, this argument goes both ways. If you can already do everything you need with SharePoint, and that is where your technical talent lies, why would you ever stick with Notes? Migrations are not about technology. They are about strategy.
I often use jQuery to bring new content into a web part, usually with AJAX calls to external systems. Actually bringing it in is not the challenge - just put your script somewhere, add a content editor web part, and call your script. The problem is a quirk of what SharePoint 2010 does with Content Editor Web Parts -- if your script runs while the page is in edit mode, it will replace your original script with the resulting content after your script has run. So your script needs to check whether or not the page is being edited, and not run while being edited. Like So --
$().ready( function() {
var inDesignMode = document.forms[MSOWebPartPageFormName].MSOLayout_InDesignMode.value;
if (inDesignMode != "1")
{
// -- Run your Script....
}
});
OK, Here is the kind of UI article that really annoys me: http://www.elezea.com/2012/09/usability-sins/ Let me sum up the article for you -- "Your users are clueless. Don't confuse them. Anything you do that they do not already understand is bad." While there truth to the idea of not confusing your users... you gotta read what was said earlier in the article, wherein he describes his audience as "tech literate". Then he goes on to say all the things his audience cannot possibly comprehend. Let's break this down. To this author, a "tech literate" user is one who:
So, if you are building web sites for these kinds of "tech literate" users, yes, his points are true. But I don't think my audience falls into these categories. I think they are actually "tech literate". UI is about knowing your audience, and tailoring their experience to what they know, meeting their existing expectations, culture, and terminology. Which means you need to understand what they actually know, not just assume that they are monkeys who are confused by blinking lights.
Sometimes, our migrations tool fail, and we end up doing some migrations of specific lists or content by hand. One mechanism to do this is to save a list as a template, including content. However, if you try to copy a list template from SharePoint 2007 to a 2010 environment, you get errors about the wrong version.
It would work, if Microsoft had not explicitly coded it to fail.
To get around this, you can simply change the version of the template file:
1) Save your list as a template, if that has not been done already. (Include content.)
2) Open your list template gallery in a browser, then open in explorer view.
3) Copy your list template .stp file to your local system
4) Rename it from .stp to .cab
5) Extract the cab to a local directory.
6) Edit manifest.xml, changing the version number from 3 to 4.
7) Create a directive.ddf file, which will define the re-creation of the cab file. This file should include the following:
.OPTION EXPLICIT
.Set CabinetNameTemplate=ListName.CAB .Set Cabinet=on .Set Compress=on .Set MaxDiskSize=0
"manifest.xml"
"filename.000"
"Add the rest of your files..."
8) from the command line, run: makecab.exe /f directive.ddf
9) Rename the .cab file to .stp
10) Upload it to your 2010 list tempalte gallery.
In short, list templates are just cab files. So you rename it as such, extract it, make the version change, and re-build the file. The settings above make sure that you can re-create it as a single file, and not have makecab.exe split it into multiple files. For small lists, you may just have one manifext.xml file in the cab. For larger lists, you could have dozens of files. But the above list is the gist of what will make it work.
There were a number of blog posts in the past 36 hours that made it clear that people feel very strongly about Lotus Notes/Domino, where it is going, what people should or should not be sharing, and how it impacts everyone involved. And while I respect the positions presented, if not always the tone, it reminded me of why I distanced myself emotionally from the Notes community a very long time ago.
Transition is difficult for most people - more difficult than they realize. No matter how adaptable you think you are, when it comes down to it, almost everyone struggles with change. And Notes/Domino is all about change these days:
There are surely other opinions out there as well, but those are the three I hear most often. I don't hear of anyone who thinks it is stagnant. Its direction may be in question, but not its motion. There are also career transitions going on, which came right on the tails of everything else. It sounds like people are going through good, positive transition, but it is transition nonetheless, so I have to wonder how much of the emotion of the past few days is coming from the stress involved in these changes, and presumably whatever history has led to these changes.
What it comes down to for me is that all of my work in Notes/Domino, SharePoint, and general web application development... it is all "just work" to me. But this was not always true. I have found that tying my own ego to any technology is a very dangerous path. It adds stress to my life, and makes me get out of whack with the whole work/life balance concept. I used to be passionate about specific technology platforms, but at the end of the day, I ended up with a split personality -- highly confident in my Domino skills, but fearful that I did not know other technologies, and lacking confidence in anything outside of the IBM realm.
Well, the truth is a nice middle ground - I am a competent programmer. I am competent in Notes/Domino, competent in .NET, competent on SharePoint, competent in Python, competent using HTML5 and CSS, and competent in JavaScript.
I don't claim competence anywhere else, but I am confident that I can learn if and when needed. And on some platforms, I am more than competent. But it doesn't help me to focus my ego on that fact. Instead, now that I am "over" the technology stack, I can actually care about the businesses I work with. I can stop fretting over tech, and start looking at what we are doing with it. And I can really listen to the needs of the business, and offer them a true consulting relationship, wherein I try to learn the business needs and drivers behind their technology, to offer them better solutions. Usually on whatever platform they already have, but also on migrations to new ones, when appropriate.
Frankly, I like it better this way. I feel more valuable, I feel like people listen more to what I have to offer, and I am able to truly engage in open, strategic discussions about what direction an IT shop should take with their technologies. I recommend that everyone try it. (BTW, the Microsoft techies are WAY worse about all this than the IBM/Lotus crowd. It was the Lotus crowd that inspired the post, but I'll take you guys over your average SharePoint or .NET zealot any day. )
My post of a week ago, combined with the upcoming end of my current contract in December, has set me to thinking. What do I want to do for my next job?
I've been watching the Lotus Notes job blog for a month or so now, and seeing mostly admin jobs or XPages jobs, neither of which really appeal to me. And it has now been a full 12 months since I last did any real Notes development. So although my career has been distancing itself from IBM for some time, I think I have to admit that the break is now truly complete. While I'm open to finding a Notes job, I just am not seeing it as a realistic next move. So then I look to SharePoint. I've said it before - SharePoint just is not fun. I'm not a fan of the technology, despite my career in it. Neither am I a fan of .NET. And I don't really enjoy server architecture, so the configuration/deployment jobs don't appeal to me either.
So what DOES appeal to me? I did have one epiphany through all my thoughts. First, let me share some of the thoughts....
So what would be my ideal next move? I want to join a large, product-based global company, that is in the midst of chaos and pain in their software/applications world. I want to be given the chance to dig into their pain points, at all levels, from front-line support to enterprise architecture and everything in between, determining root causes of the pain, and defining the actual business drivers behind all the applications and technology. I then want the authority and resources to devise and implement fixes to the chaos.
I am not sure what the job title would be. I could see this role being a director-level position, setting roadmaps and guiding a large organization... but it could also be an individual contributor role, focusing on key areas that are critical to the business. I would prefer the leadership role.
My next steps actually become clear at this point. Now that I have focused my goals a bit, I can stop looking at job boards, and start instead asking around to find companies that fit my scenario. I can stop sending resumes to HR, and start having frank discussions with IT leaders about how I could personally help them. I'm not sure exactly where I will end up, but hopefully come January, it will be somewhere satisfying.
About a year ago, I started to be very disillusioned with Facebook. I always had just used it to keep in touch with old friends... I had a "No App" policy, etc. My reasons for disliking Facebook are not unique - Privacy issues, the superficial nature of the contact with other people, etc. But as politics began to creep into people's posts, I noticed something that inspired me to shut it down entirely, and delete my account.
Let me start my saying that it is completely natural for old friends to grow up with different opinions than my own. But when you are sitting face to face with friends, you know what to talk about, where to steer clear, and you can see reactions to know when you have crossed any lines, and take steps to improve your communication in ways that smooth out any rough edges, and have a true discourse on a variety of topics, without damaging friendships.
Facebook takes all those controls away, and you get raw opinions thrown in your face. It was disconcerting seeing things that I really disliked coming out of the mouths of friends whom I hold nothing but positive feelings for. And that was the spark that started to drive me away. It wasn't the entirety of my reasons, but it did set me to thinking... And I realized that I probably would not be close friends with all of these people today if we still interacting on a daily basis. I didn't like that. I got the sense that I was sometimes better off asking, "Gee, I wonder whatever happened to that guy/girl...", and not knowing, than to be able to just pop on Facebook, and with no mystery at all, have it laid out right before my eyes.
Naturally, this was not true of everybody, but again, it was a spark of thought... So I asked myself - what do these connections truly add to my life? Am I truly better off seeing occasional posts from the girl I took to my Junior homecoming dance? Or the boy I used to walk home with in 3rd grade? And my answer was "No." Not only did these connection not really add anything, they diluted my positive memories of them.
I also found myself wanting to share my daily experiences with old friends via Facebook, and I stopped sitting down at the end of the day and sharing things with my wife. When we would all come home after a long day, we didn't sit down and play cards together anymore... we sat down at our computers. We had not even gone outside and taken an evening walk in over 6 months!
This is not a good way to live. So, I exchanged email addresses with the people whom I really am friends with, connected on LinkedIn with people whom I knew through work, and deleted my connections to everyone that I actually see in real life, forcing me to... well, to actually see them in real life. After a couple months to be sure I had a new way to contact these people if I ever wanted to, I deleted my Facebook account.
That was 4 months ago. Since that time, I have come to similar realizations about many online activities. I deleted my account on Reddit. I deleted my account on Hacker News. I make a conscious effort to come home and NOT get on my computer. I am still the first one awake in my family, but once I hear someone wake up, I go upstairs, and frequently end up wrestling with my sons, talking to my wife or my daughter, and sometimes wrestling with my daughter, talking to my sons, etc. The activities vary, but I spend almost every weekend morning just hanging out with my family. We go outside, we ride bikes, we take walks, we go to the park, we have picnics, we play in the backyard, and we have campfires and roast marshmallows. I go out to lunch with people and engage in truly wonderful conversations. I call people on the phone. I have even thought about writing letters, but so far that is just a thought.
The point is that I refuse to believe that ANY content on Facebook or reddit or any other web site is better for you, or your family,or your friends, than spending time together. I am so grateful that I am changing my habits. And I'm hopeful that over time, more people will disconnect just like I did.
So, the following scenario will cause an error when working with sites and site templates:
The template now cannot be de-activated, and therefore cannot be deleted. Instead, you will get an error along the lines of:
"Unable to access web scoped feature because it references a non-existent or broken web"
If you Google this error, you will find people telling you to restore the site first, then delete the template, or giving you powershell commands to try. But none of these made sense to me - why in the world would a template REQUIRE an active site based on itself to allow its own deactivation? If anything, it should be the exact opposite.
So I checked the site collection recycle bin, and saw the site I had deleted sitting in there. So I emptied the recycle bin.
Voila! Now I can deactivate and delete the site template.
So, should anyone hit this error in this scenario, just be sure your site is actually gone from all recycle bins, and then your template is free to deactivate.
Ok, here is another post by someone who comes off as just... wrong. http://blog.capwatkins.com/recruiting-a-tutorial
Now, I'm not going to defend the recruiting industry. Many recruiters are not going in depth to find good matches between people and jobs. But I also am not going to write off the whole industry that way.
See, I love recruiters. I have had recruiters find more than half of my gigs since the 90s. Sure, there are some (quite a few) bad apples out there, but there are also some truly awesome people who will find great work for you, if you give them a chance.
What really shocks me in this post, though, is the attitude that phones are private. It really was not that long ago that phones were the primary way that all people got in touch with each other. And caller ID used to be rare, too. Every time the phone rang, you had no idea if you really wanted to talk to the person on the other end or not. And yes, telemarketers were a huge annoyance.
So you know how to deal with this? Don't answer the phone if you don't know the number. If it is important they'll leave a message. It is called screening your calls, and not letting your phone run your life. I don't take a lot of cold calls either, mostly because I don't answer. But to get angry and hang up just because they got you on the phone? To write a blog post that just smells like a huge superiority complex? That is juvenile. I normally tell recruiters who call me the following pieces of information:
I can tell you that 90% of people never actually keep track of me and email me back. But the 10% who do are the good recruiters who you really do want to work with, because they just proved that they are listening to you. All I am doing is turning a cold call today into warm leads in the future. Whereas all you are doing by telling these people off is exactly that - turning them off so they won't call you in the future, they will call me instead. Well, hang on a sec -- based on that, sure, feel free... tell the recruiters off. More work for the rest of us.
Hey All,
Check this out -- http://techpinions.com/windows-8s-greatest-sin/11907
It covers the underlying reasons that I suspect Windows 8 is going to usher in a few years of very busy IT work. The article focuses on the consumer level, but the same impact will happen at the enterprise IT level. It shops need to either stagnate or migrate.
Migration will involve strategic choices to go all-in on Microsoft technologies (as Windows 8 at the desktop only works if you are also up to the latest versions of your servers), or to start to move away from Microsoft.
Which doesn't mean that I am predicting Linux to take over the world, nor do I predict Microsoft to start to lose a ton of market share (I do predict some losses, though.)
What I am really predicting is that were about 18 months away from a 3-5 year bumper crop of major IT Transition projects, no matter what platform IT shops are going to. This will not be just the front-end application migrations that I have been talking about for the past 5 years. It will be full-stack migrations to new versions of everything.
We're all going to be extremely busy from 2014-2016.
I had a frustrating conversation with a recruiter over the last few days. Let me be clear, this is a recruiter that I like, that I have worked with before, and who has always been the good kind of recruiter - one who listens to you, suggests matching positions, keeps in touch even over multiple years, so she is available when I want to explore new options, etc. I would recommend her to anyone, and although this specific conversation is frustrating, this is not a dig at her personally. She rocks. Yet somehow, we ended up in an odd spot. So I wanted to share this simplified, and exaggerated description of how the conversation went:
Her (via a LinkedIn post): Hey Everyone, I have a startup client looking for JavaScript talent.
Me: Sounds interesting. I'm not up to speed on backbone.js, etc, but I know JS well enough to pick it up. Tell me more?
Her: It is a startup run by some founders of Sun, here in Denver. They are well funded and growing fast. Interested?
Me (to myself): The founders of Sun? Aren't those the guys who were suing coffee shops in the 90s because the coffee shop names had the word "Java" in it? Those guys? And didn't Sun eventually crash and burn? Whatever, I'll go research them. Aha -- they are writing software to enable social conversation... on Twitter? Is this a joke? And they are selling it to Enterprise IT shops? Why? I think collaborative software for the Enterprise already exists... Oh, well, these guys have built companies before, they are well funded, so I'll give them the benefit of the doubt until I really know what they are doing. And hey, maybe a startup in a nice downtown office would be fun. I'll at least be willing talk to them.
Me (more professionally, to her): OK, their product makes me a bit nervous, but they do have experienced leaders, so go ahead and give them my name. Here is a resume updated to show more of my JavaScript experience.
Her: Thanks, but my boss thinks you look like more of a manager than a coder. Can you update the resume to make yourself look more like a coder? They are trying to meet a deadline and are desperate for this JavaScript talent.
Me: Frankly, No. My broader experience should be an asset, not a detriment. If they disagree, no big deal. This is not a match. Let's move on.
Me (to myself): "Desperate" is a bad sign. How did they get to that state? What is the real story going on here? And now that I think of it... what does this imply? Are they really doing a startup and trying to pigeonhole early employees into strict roles? And wait one more sec - if they are close to missing a deliverable, shouldn't they know better than to just throw more developers into the mix? That is going to slow down delivery, not speed it up. And while I am thinking about it, why is a startup already at 50 employees, and hiring 40 more next year? This sounds like some theoretical hiring model that was designed before they started product development... they have the funding to roll with it, whether or not the product development is matching their original plan. Whoa, hang on a sec -- these guys are sounding scary, actually. They seem to be forcefully driving to a plan instead of reacting to reality. Maybe they are not such a hot leadership team after all...
Her: Yes, actually I agree. My boss just want to show more technologies on your resume before sending it on to them.
Me: Understood, but no. My resume is honest, so take it or leave. I'm fine either way, Really.
Her: OK, thanks!
Like I said, she listens and she gets it. But I don't think her boss does. And the 3rd hand info about this startup makes it sounds pretty sketchy actually. Well-funded, popular with the media, could be a nice poster child for the Denver startup scene, but kinda scary from an actual professional perspective.
See, I've fallen for the trap of the shiny new startup before. You go in all excited, you honestly think you are going to come out with a successful product, and you commit yourself fully to this effort. But it always comes down to the leadership. Every single startup I have done has succeeded or failed based on the leaders. And they mostly fail.
The startup community can pontificate all day about how to do startups, but there is really only one answer - have a leader who is worth following. And that is what i am holding out for, for any permanent job -- I want to follow a great leader, and be part of a great organization. So I don't feel bad at all that my friendly recruiter's boss is nixing me because my resume doesn't fit their mold. That in itself is a red flag, but there are SO many other red flags in this conversation... I'm finally old and wise enough to actually let the red flags stop me, and happily walk in the other direction.
The inspiration for today's post: http://johnjardin.ukuvuma.co.za/2012/11/09/use-dojo-or-jquery-to-manipulate-printing-of-web-pages/ Sure, that is a clever little trick in jQuery. But totally not necessary. Easier solution:
@media print { .noprint { display:none } }
I use jQuery extensively every day. But some things just don't need it. Try the easy answers first.
I was looking at a project that I did last year, and realized that there are parallels to certain techniques within Domino development to the MVC design pattern.
MVC is very popular in web frameworks these days, and I tend to like it. I won't re-hash a definition of MVC here, but if you are not familiar with it, I suggest you check it out.
In any case, in a Domino app, we can define the Form as being equivalent to the View - they both format a data set for presentation in a browser. We can write an agent that is equivalent to the Controller, pulling data from the Model, performing business logic operations against it, and send the results to the View. The mechanism for doing so in Domino is well served by using a WebQueryOpen - enter your "Controller" agent in the WQO of your "View" Form, and you have just created a Domino app that follows the spirit of the MVC pattern.
While this is obviously just one of many way to create a Domino app, I found it extremely useful when aggregating data from multiple Domino documents (or databases) into a single web page. It also will bring your domino apps in line with the design of other modern web frameworks, making a transition to another web platform a bit easier.
I sure am getting in the habit of reacting to other blog posts more than writing original content, but hey, it is working for me, so here is today's:
http://blog.texasswede.com/export-from-notes-to-excel-3-different-ways/
The pertinent quote that caught my attention was:
But if you just need to export raw data to Excel, there are a couple easy ways to do that. You don't need to automate Excel, something the programmers posting in the forums seem to frequently do. That just makes things more complicated, and if you run the code in a server agent, Microsoft Office have to be installed on the server itself.
I read this, and immediately thought: "Aha. Excellent point!".
I always have been in the habit of coding to the Excel objects, which does require a local Excel install, and does require Office on the servers. I freely admit that I got stuck in my habits from eons ago, and never updated my thinking.
He is right. My ways are old-fashioned, and his ways are better. I just wanted to call out his post, and thank him for knocking me back on-track in my own thinking.
I'm going to be starting a new job at the first of the year.
A Web development job, back on Domino. There is no SharePoint in this job. I am so grateful to be getting away from SharePoint.
While most people know that I think Domino will continue to shrink in the enterprise market, this is a different story. A small, talented team who can handle the platform, combined with a product for which they have the flexibility to choose their platform, regardless of Domino's market share.
I'm not sure if I'll have anything to say here in the future or not. I enjoy posting occasionally but I also tend to get somewhat unprofessional at times, which is a good habit to break.
And doesn't it sound like a nice conclusion to this 5 year saga, to be able to say that at the end of it all, I am walking away from SharePoint, and back to my roots, developing solid applications on Domino?
One disadvantage of suffering from chronic pain is that you don't sleep as much as you want to. But that is balanced with having a lot of time to think. And I was thinking this morning about the prior startups I have worked for, and some of the lessons I learned from each. So here they are (names withheld to protect the innocent):
1998-2000 - Content Management Startup, Denver, CO.
We had a few good development contracts, and things were looking promising, so we expanded our company - opened offices in new cities, hired more staff, expanded our marketing efforts, etc. However, the business did not come together the way we had hoped, and the majority of the revenue for the company came from a single customer. When that customer started to fade, the company was in trouble.
Lesson - don't base your company strategy on the revenue from a few large customers. Without a diversified customer base, you are at major risk in the case that your large customers go a different direction. Base your strategy on the possible implications of risk.
2001 - Content Authoring Tool Startup, Oregon
I spent 9 months writing authoring tools in Flash, which at the time was used for nothing more than annoying web site designs. I wrote re-usable components to build UIs for business applications, drag/drop UI editors that would export HTML for site designs, and tools to tie these UIs to back-end web applications. Pretty good stuff for 2001.
Lesson - Don't work in a vacuum Once I had my complete toolkit ready to ship, Macromedia released the next version of Flash, and the new version contained about 75% of the features that I had just written. Had I been paying attention to their direction, I could have skipped most of the development work I did, greatly enhanced the other 25% of my product, and had a great launch of complementary tools to go along with a major release of Flash. Oops.
2002 - Content Management Startup, Denver, CO
Differently flavor on the same product. But this time, it was a small Denver shop, owned by a large German consulting firm. It felt like we were doing great, adding new customers, always engaged in projects... hiring more people... right up until they fired half the company, and the CEO was begging the German parent company to not shut us down.
Lesson - be reasonably transparent in how the company is doing. Us development folk were really frustrated with how the problems were handled. We could have done more to help if we knew what the problems were. Instead, we got blindsided on multiple levels, and lost a ton of trust in the CEO. Small companies need to work together and trust each other.
2005 - Real Estate Startup
I took over development for a real estate startup, providing web sites to brokers in a SaaS model. Unfortunately, the prior dev team left us with a product that did not scale, which led to customer service disasters, and we spent all of our time trying to resolve customer issues instead of developing our product. The founder of the company had offered unlimited free support to his prior customer base in exchange for pre-payment of the monthly fees for a year. That money was supposed to fund the development of the product. It instead just funded the aforementioned customer support. We did complete the next version of the product, but we had already run out of money, and the owner refused to pay the development staff. You can guess how long the company lasted after that.
Lesson - Your business model matters, often more than the technology. A great product that is set up under a bad business model can easily fail.
Overall Lessons -
The product developed for a technology startup is a base requirement to get a company going, but once it exists, other factors become critical. You also need to have a solid business model to operate under, and a market to operate in. Leadership of the organization is critical, in their trustworthiness, their understanding of how to lead an organization, their business sense, and their adaptability.
So the error I was receiving was "invalid token o". OK, fine, I have some bad Javascript syntax somewhere... this should be an easy fix, right? Except that all the scripts looked fine. I wasted almost an hour methodically checking all the scripts for an extraneous character, a missing semicolon, etc. I killed domino's _doClick function, as it has an 'o' object, and I'm not using it anyway. Nothing was changing my error. Until I figured out the problem - the error was coming up on a callback from a jQuery AJAX POST. My normal mechanism on these is to parse the returned data string into a JSON object, then do my work against that object. However, this particular AJAX POST was already returning it as an object. So when I did my "JSON.parse(data)", it converted 'data' into "[object Object]". And guess what? That is not valid JSON! So guess what, that 'o' is an invalid token! Yeah, an hour of futzing around only to realize that I didn't need to parse that object.
Thomas Duff has started a discussion that falls well in line with my original intent of this blog:
http://www.duffbert.com/duffbert/blog.nsf/d6plinks/TADF-94Y7KP
For 5 years, I had a healthy mix of both platforms. In the 2nd half of 2012, I did not touch Notes for 6 months, while working a 100% SharePoint contract. Now, I have a permanent job in a Notes shop, where SharePoint does not exist, so I am not even thinking about SharePoint.
And you know how I feel about the discussions of Notes vs. SharePoint now? Happy to NOT be part of it. I tossed my one and only piece of advice to the community out into the comments, to be taken or left as you will.
But so much of Notes vs. SharePoint gets caught up in emotion, politics, marketing, and money... not just from the community, but from the product management coming out of both IBM and Microsoft.
I cannot be an evangelist for either platform. I am competent in both platforms, and I evaluate the platforms on their own merits (and faults). I am happy to work with whatever platform will meet the needs of my organization.
I sense that most Notes folk who move away from Notes end up in this same place -- more technology agnostic, looking at technology platforms simply as toolkits, and not letting any one vendor become part of your own self-identification.
And really, when people get to that place, work environments improve. People work together to reach common goals, not personal agendas. It is a better place. A happy place. A lot of us are already there. We hope the rest of you can join us.
Any of you who know me know that I am against migrating anything to XPages. And there is a frequent misunderstanding that I dislike XPages as a technology platform. I don't. On the contrary, I totally see its appeal, and see how you could do great things with it, quickly.
No, The root cause of my resistance to XPages is because work done on XPages is tightly tied to the back-end platform.
After a couple decades working as a software developer, migration to new platforms is a simple fact. It might be next year, it might be in 10 years, but you will NOT be on the same platform forever. You will migrate somewhere, someday.
And the one consistent fact for the past 20 years, and for the foreseeable future, is a front-end based on HTML. Maybe it will be browsers, maybe it will be tablets, phones, or something else, but HTML, CSS, and JavaScript are consistent across a vast majority of present and future platforms, and I see no indication of change in the near future.
So I want as much of my code as possible to fit into this front-end paradigm, or at the very least to be back-end code that puts the generation of the front-end code into my own control. Then we can rip and replace back-end systems, while still maintaining a large quantity of our codebase, because our front-end layers work regardless of platform.
And therein lies my resistance to XPages. Every time I see a blog post about "How to do this in XPages", it reinforces my notion that if XPages needs a separate set of instructions vs a generic "Here is how to do it in any web page", then something is wrong. It always kind of feels like, "Here is something that the rest of the web development world is doing, and here is how to make it work in our weird little corner."
This is not unique to XPages. I dislike SharePoint for the same reasons (among others).
It comes back to the basic principle of separation of layers. I prefer platforms with a clear mechanism to split UI/Data/Business layers. This split decreases future development or migration efforts. And that is good business -- When doing a large new project, you often have dedicated time set aside to get the initial development done. Down the road, the application will be well established, and will need more agility and flexibility. Separation of layers achieves this for you, as you can replace discrete potions of the app, without an entire re-write.
So do we spend more hours to achieve this using old-skool Notes development? Probably. Is that extra effort worth it over the next 5-10 years? Definitely.
It all boils down to a very long-term vision for me. Are you an IBM business partner, where your business goals are tied to IBM's success? Then you have a vested interest in XPages. The rest of us have the opposite interest, in maintaining our long-term flexibility, to allow us to respond to changes in our business.
One final word - I placed all of this in the context of XPages, due to the Lotus-centric nature of this blog. But the principle applies across all platforms - give me separation of layers, and give me control over how my front-end HTML is generated.
My main problem with the "startup" world is the huge blinders they wear around themselves. They think everything they do is new and innovative, while they proceed to create minimal variations on existing technology, with minimal positive impact to the world around them.
This morning I saw the latest example of this:
http://www.businessweek.com/articles/2013-04-10/silicon-valley-goes-hollywood-top-coders-can-now-get-agents
They are acting like this is a revolutionary idea -- coders can have their own agents, just like celebrities do!
No, this sound a lot more like the standard recruiter. They find jobs, negotiate payment, handle billing, and take a cut. This is not innovative. This is just a new label on a very established industry, but because they are doing it smaller, they are claiming it is something new.
Sorry, folks. You just reinvented the same old recruiting firm.
At one point, I had some choice rants on this blog. I started being more cautious about my tone when I left my job 2 years ago, and thereby could also lose my anonymity. For better or worse, it is easier to go on a rant when people do not know who you are.
But right now, my stress level is through the roof (for non-work related reasons), my insomnia is also problematic, etc, etc.
So maybe it is time to go ahead, let go, and toss a rant out there. I may regret this tomorrow, but it sure will make me feel better today.
So here goes:
I am REALLY tired of web developers online becoming extremely arrogant and self-righteous, in the guise of helping others online. I see people post some old code they inherited, and mercilessly tear it apart. Post some sample code that isn't perfectly semantic, and they treat you like you are an idiot.
Well, do all you young, perfect, web developers know where the best practices came from? Do you know how the concepts of semantic code have come from?
They are lessons learned by those of us who have been doing it for 20 years. 20 years of just trying to do good work, hitting problems, seeking better solutions. 20 years of Microsoft, IBM, Mozilla, Netscape, AOL, Time Warner, Walmart, Kentucky Fried Chicken, and the Evil Bunny trying to get everyone to do the web their way. 20 years of browser and server technologies getting ripped and replaced, reinvented, and rethought. 20 years of 2 steps forward, one step back, across multiple industries, with slow incremental improvements after each cycle.
Today, we have made great progress. We have semantic markup, we have MVC frameworks, we have scads of choices in technology stacks to use. We have the internet to do all the research, learn best practices online, and computing resources that would have been unimaginable 20 years ago, on which anyone of any age can practice and learn.
We have handed this all to you young folk on a platter. Think about learning web development before the web was this pervasive in our world. Think about having to learn from actual books, training classes, Usenet groups, and good old-fashioned experimentation. Think about there being no experts posting all their knowledge and wisdom online for your casual perusal.
So forgive us old farts if we sometimes fall back into old habits. Forgive us for doing our work the way did, say, 5 years ago. Or even 10. We learned ways to make things tick. We know how to write apps that will solidly meet business needs, with little maintenance required.
Maybe we do overuse onClick events. (Go ahead, say it... "onClick events are NEVER semantic!") Maybe we do fall back and throw in a a table-based layout to get a quick job done instead of fighting CSS floats and clears for 3 hours. And yes, maybe we don't even "get" some of the newest technologies, practices, and paradigms.
But maybe our code still works. Maybe we actually spend 40 hours a week coding, instead of 10 hours talking about it on Hacker News or Reddit, 5 hours browsing code on Github, and 7 hours trying to come up with the next killer social media app, leaving only a few hours a week to do something useful, like YOUR ACTUAL JOB. Maybe we develop loyalty to employers and leaders who treat us well, instead of being mercenaries just trying to "monetize" every line of code we write.
And maybe... just maybe... we were once 24 years old and brilliant and perfect once, too. Maybe we have been where you are. Maybe we were the ones who created the original Internet Bubble in the late 90s, and maybe we survived it, and have been working steadily for almost 15 years since then. Maybe we learned some lessons there. Maybe we have decided that there is more to life than the value a Venture Capital firm puts on your latest idea.
So maybe... it is worth forgiving us when we write "bad" code.
Hey, you very well might be smarter than me. You might be a better coder.
But whether you are or not, if you are someone who enjoys ripping other people's work to shreds because it doesn't perfectly meet your own standards... I really have just one thing to say.
LIGHTEN UP.
I'm now going to take my own advice, chill out, and go write some bad, schlocky code that would make you cringe.
In response to: https://medium.com/on-coding/b108eee5aeee Where they conclude:
I think we should all have a rudimentary understanding of our pervasive technology and how it works, so that we can make more intelligent decisions.
While I agree with that statement, I wonder what makes people think that coding is the most obvious way to apply that thought... How about food? Do you know how to grow it? Hunt it? Do you know commercial food is grown, transported, and packaged for sale? How about toilets? Do you know how your plumbing works in your home? Can you fix it yourself? Wouldn't that be a more critical system to understand than your mobile phone?
But even aside from those examples, lets go ahead and talk about having an understanding of technology. Do you really understand how the cell phone networks function? How your mouse actually works? How your wireless router knows which computer to send a packet to? (Trick question, it doesn't, which is why people can hijack your data in coffee shops. See how not knowing things can hurt you?)
My point is that we have many more critical systems in our society than mobile phones and web apps. The author of the article is arguing that code teaches you analytic thought, and it teaches you to break down problems into smaller chunks, helps you interact with world, and gain an understanding of many topics. And I do agree that coding can help develop analytic thought, and can open worlds of creativity. I won't stop anyone who wants to go there.
But coding is only one way to get there. Not the only way, and I don't think it is even the most efficient way. I think it is a prerequisite to coding, not a result of coding. But some people learn them at the same time, and it may cloud the cause and effect.
I'm less interested in arguing about the nature of thought than I am interested in being sure out world maintains its diversity -- there is so much more going on in our world than technology... I want artists and musicians in our world. I want park rangers, mountain climbers, and photographers. I also want plumbers, electricians, and someone who can fix my leaking roof. I want a huge variety of people in this world to find their own skills and their own places.
I have always really liked the idea of Kickstarter. REALLY liked it.
But I do not like how it has actually come to fruition in reality.
It seems like people are now tossing half-hearted ideas out there, with an attitude that they might do something, if people pay them for it. Or that kickstarter is a funding mechanism for a business project, or a startup.
I greatly prefer the idea of it being truly a mechanism to offer funding to people who are going to do projects anyway, and the kickstarter funds are a means to overcome financial hurdles.
I used to see people put hours of effort into indie games, for example, and maybe need some funding to push to the end. Now I see those efforts going into marketing for the kickstarter campaign, and not one bit of work is done unless it is pre-funded via the kickstarter campaign.
This is wrong, people - creative projects should be something you are doing because you want to do the project. Using kickstarter as a way to just get paid for a job that you would not otherwise do... Well, that is boring, and it dilutes the creative value of kickstarter.
It is no secret that I do not like where the "startup" community has gone. Everything is about money. People are developing their ideas, not to be creative, but to make money. That is not what creativity is about. That is not passion. That is just greed. And using kickstarter as a vehicle to satisfy your greed is kinda lame.
There are good creative projects out there, both on kickstarter, and not. I just have this dream where people will go back to being creative for its own sake, and let money become simply a tool, not a goal in and of itself.
Github is just a code repository, but programmers frequently use it as the default "home page" for a project when sharing it around the web.
The problem with this is that the default Github UI is to first take up your entire screen with the folder/file structure of the code. While this makes a ton of sense if you are a programmer on the project, it makes no sense whatsoever if I have never seen the project and just want to know what it is.
Most of the time, people do share what it is, but the UI is backwards. It is usually first a display of the code, then technical details about the code, THEN maybe the high-level explanation of what it does, and sometimes a link to a demo.
When sharing a project with the world, this needs to be reversed. First tell us the high-level purpose of the project, then a demo, then tech details, and finally code. I know Github supports this - I have seen it done.
Now, I foresee a complaint here - that may work for the general public seeing the project for the first time, but it makes the actual programmers scroll around, and they are on the site more.
At that, my friends, is the real problem. Github is a code repository. Nothing more, nothing less. Trying to also use it for marketing is a losing battle.
I remember when I first started to write code, and I was struggling with basic stuff, as all new programmers do, I was told a story about some old programmer at Apple. The gist of the story was that this guy would just sit in his office for hours, staring at the ceiling, thinking about what he was going to code. After a couple hours, he would sit up straight, grab the keyboard, and bang out the code very quickly, as he had thought it all out.
I recall thinking that such an act sounded so difficult and it was very alien to how I worked, which was to code every detail as I thought of it, and eventually build it up into an app. Now that I have been doing this for years, my personal workflow is very different, and I can understand how a coder could stare for hours, then type it all out.
I am not quite like that, though. I leave my desk a lot more. I walk away from my keyboard to think about exactly how I want to put something together. There is this thought process that goes on as you break business requirements down into smaller functional chunks, mentally apply all the coding techniques and platform features to each component to figure out what will work best, and then merge all the components in your head, making sure your design is solid.
I tend to run through a variety of designs as I think something through, and eventually decide on the high-level structure of the application. At that point, I sit down, and knock out the structure of the app. I then knock out the details at my computer, iterating through the small details, coding and testing as I go, and spend my hours at my computer finishing my ideas into a working app. And I have found over the years that I really do need to be away from the computer to go through that first 50%.
Doing something physical, but mostly mindless, is how I work. 10 years ago, I used to go on a hike when I had something big to think through. But I ended up halfway up too many mountains, suddenly having everything click, and have to turn around and hike 2 hours back to my car to go code it up. Bike rides work better, as they are shorter. But I've found my ideal environment is just to be around the house, so I can jump right into the flow of coding once things "click".
I may clean the house, work on art projects, sometimes play a game or two, etc. Which has made my wife laugh at times, as this method of working doesn't really look like I am working. I may get off a conference call and have a pile of new work, or changes to existing work. What is the first thing I do when I get a new pile of work? Walk away from it. What do I do when I hit a big problem, or a tricky bug pops up? Walk outside and go do something else.
Counter-intuitive it may be, but it works for me - sitting at a computer, beating your head against a design, or a problem, or a bug... just wastes my time. It burns me out, and doesn't speed up the thought process. Walking away for a while,spending about half my brainpower pondering it, while doing something unrelated with the other half? That works. I frequently come back feeling rested, refreshed, and with the right answer. Sometimes that happens during normal business hours. Sometimes I wake up at 2 AM, code until lunch, then sleep in the afternoon. And sometimes it isn't even a matter of a deep and complex thought process. Just last night, I had a javascript function that just would not work. I could not see why it was failing. So I went to bed. Got back up at 3 AM, looked at the script, and immediately saw that I was missing a character in some element ids.
The point to all this is that coding is a very personalized process. Some people can work 8 hours straight, some cannot. Some people work effectively in an office, at a computer 100% of the time, some do not. I currently work from home. It works well for me, as I can do what I need to do to deliver work, without fretting over anyone judging me for when or how I work. There are days where I write almost no code, but think through all the options, and then have another day where I code for 10 straight hours. Sometimes I have a very productive day/week, and can write almost an entire product overnight. Other days/weeks, not so much, and I think to myself, "Oh, I need to get this week on track before they fire me from this awesome job." The ideal environment for coding is much discussed online, but it comes down to self-knowledge, and finding a boss who lets you do what works for you.